AI Dataset Version Manifest: Sources, Labels, Splits, Rights, Hashes

An AI dataset version manifest is a machine-readable record that binds one immutable dataset release to its files, source lineage, collection window, rights evidence, label policy, transformations, train-validation-test split, quality reports, limitations, owners, and cryptographic hashes. It works beside a human-readable data card or datasheet. The card explains why the dataset exists and how it should be understood; the manifest lets a pipeline verify exactly which approved bytes and rules were used.

Use public, synthetic, or properly governed data while learning. Do not copy personal, copyrighted, confidential, scraped, licensed, or client data into a project merely because it is technically reachable. Data rights, privacy, consent, security, and sector rules depend on the source and intended use. Route uncertainty to qualified owners before collection or release, and keep credentials and raw sensitive records out of prompts, screenshots, repositories, and general logs.

Define The Dataset Decision

State the task the dataset supports, intended users, unit of analysis, target population, labels or outcomes, geographic and time scope, and decisions a model may influence. Name the dataset owner and release approver.

Do not begin with collect everything. A bounded purpose determines which records are necessary, which uses are prohibited, and which evidence reviewers need before release.

Assign A Stable Identity

Give the dataset a durable name, namespace, version, release date, owner, and canonical manifest location. Distinguish a dataset version from an experiment run or model version.

Use immutable release identifiers. A latest alias may help discovery, but training and evaluation records should reference the exact approved version and manifest hash.

Inventory Upstream Sources

List every database, API, file, survey, public archive, generated source, and manual contribution. Record source owner, access method, collection period, selection rule, expected volume, and upstream version.

Google Research’s Data Cards work emphasizes upstream sources and the processes that shape a dataset. A generic public web label is not enough provenance for rights, quality, or reproducibility.

Document Rights And Restrictions

For each source, record license, permission, consent basis, contract reference, attribution, geographic restriction, purpose limit, retention, deletion, and redistribution rule. Store protected evidence outside the public manifest and reference it by controlled identifier.

Do not infer permission from open access. Public visibility, copyright license, privacy authority, consent for AI use, and permission to redistribute are separate questions.

Minimize Sensitive Data

List fields collected, why each is required, sensitivity classification, and who can access it. Remove direct identifiers and unnecessary attributes before wider processing where policy allows.

Pseudonymization reduces some exposure but does not make data anonymous automatically. Preserve re-identification risk, linkage, and deletion obligations in the data card and access policy.

Describe Collection

Record collection code version, query or sampling frame, inclusion and exclusion criteria, dates, locale, devices or channels, rate limits, and known outages. Save a hash of configuration and code used.

If collection changes midway, either create a new source segment with documented differences or issue a new dataset version. Do not silently blend incompatible processes.

Freeze The Raw Snapshot

Store an access-controlled immutable raw snapshot or a verifiable source reference according to policy. Hash each file and a canonical file inventory. Record byte size, media type, row count, and schema.

Never modify a released raw file in place. Corrections should produce a new version with a change record, even when a filename stays familiar to operators.

Specify The Data Schema

Define every field’s name, type, unit, allowed values, null behavior, semantic meaning, source, sensitivity, and transformation owner. Reject unexpected columns and incompatible types during build.

A CSV header alone is not a schema. Capture category dictionaries, timezone, encoding, coordinate system, language, and nested structures where they affect interpretation.

Write The Label Policy

For supervised data, document label definitions, evidence requirements, precedence, ambiguous cases, abstention, and escalation. Include positive, negative, and boundary examples.

Measure reviewer agreement and preserve adjudication rules. A clean label column can conceal unresolved disagreement that limits the model’s achievable performance.

Record Annotation Operations

Identify annotator qualifications, training, tools, compensation or role, assignment method, blind or assisted review, quality sampling, and adjudication. Protect annotator privacy.

If models propose labels, record model version, prompt or configuration, confidence use, and human-review policy. Machine-assisted labels must not be presented as independent human ground truth.

Version Transformations

List deduplication, filtering, normalization, resizing, tokenization, redaction, augmentation, and feature extraction in execution order. Reference code version and parameter hash for every step.

Separate reversible normalization from information-losing transformation. Keep counts before and after each stage so unexpected loss or amplification is visible.

Control Duplicates And Leakage

Define duplicate keys and similarity rules, then report exact and near-duplicate counts. Check repeated people, documents, sessions, templates, and derived records across intended splits.

Leakage can make evaluation look excellent while measuring memory or shared context. Perform split checks after every transformation that can create or reveal duplicates.

Create Deterministic Splits

Document the split unit, grouping key, time boundary, stratification, random seed, algorithm, and target proportions. Hash the membership list for train, validation, and test.

Split by the entity that must generalize. Random rows are unsafe when the same person, product, conversation, document, or future information appears in multiple partitions.

Protect The Test Set

Restrict who can inspect final test labels and how often the set is evaluated. Record each authorized use. Repeated tuning against the test set turns it into development data.

When the test set is exhausted or the operating context changes, create a new governed evaluation version rather than quietly replacing records under the same identifier.

Run Quality Checks

Validate schema, nulls, ranges, file integrity, decoding, label distribution, duplicate rates, split isolation, class balance, timestamp order, and source coverage. Store machine-readable results and thresholds.

Do not collapse all quality into one score. A dataset can pass technical checks while still having unsupported rights, weak coverage, harmful categories, or poor documentation.

Review Dataset Security

Threat-model ingestion, storage, annotation, export, and training access. Scan files for unexpected executables, active documents, malformed media, secrets, prompt-injection content, and archive expansion risks before they enter trusted systems.

Use least privilege, isolated processing, approved file parsers, size limits, malware controls, encryption, and auditable access according to the data class. A valid hash preserves malicious bytes just as effectively as safe bytes, so integrity and security are separate gates.

Analyze Representation

Compare dataset composition with the documented target population and use context. Report known undercoverage, overcoverage, temporal gaps, language gaps, device effects, and proxy variables.

Avoid claiming representative without a defensible sampling frame. Small groups require careful disclosure because unstable estimates can mislead or expose individuals.

Document Intended And Prohibited Uses

State supported research or product tasks, required preprocessing, unsuitable decisions, prohibited populations, and environments not evaluated. Connect limitations to practical user choices.

Datasheets and Data Cards are designed to communicate purpose, composition, collection, use, and evolution. Keep restrictions near the manifest so they travel with the files.

Add A Machine-Readable Inventory

Include relative path, role, media type, size, checksum algorithm, checksum, row or item count, schema version, and encryption or access class for every artifact. Sort entries canonically before hashing the manifest.

Use a modern organization-approved cryptographic hash such as SHA-256 for integrity verification. A checksum detects changed bytes; it does not prove that data is accurate, safe, or authorized.

Sign Off The Release

Require review from dataset ownership, domain, privacy or security, rights, annotation, and model evaluation roles according to risk. Record decision, exceptions, owners, expiry, and rollback or withdrawal route.

The AI project data-readiness checklist can help prepare evidence, while this manifest freezes the approved release. Do not approve with missing source or rights records.

Verify Before Every Run

At pipeline start, resolve the immutable dataset version, verify manifest and file hashes, validate schema, and record the manifest hash in the experiment. Fail closed on missing or unexpected artifacts.

Do not download latest during a reproducible run. Cache or mirror only under approved policy and verify the bytes before training or evaluation.

Track Changes And Withdrawal

For each new version, list added, removed, corrected, relabelled, and re-split records with reasons and impact. Preserve prior manifests according to retention policy.

When consent, rights, or quality problems require withdrawal, mark affected versions, notify approved consumers, stop new use, and document whether derived models need assessment or retraining.

Monitor Downstream Use

Record which approved models, experiments, reports, and products consume each dataset version. Capture feedback, incidents, discovered biases, and limitations from downstream teams.

The Artificial Intelligence course can build foundations in data preparation, models, and evaluation. Production stewardship still needs accountable ownership, access control, documentation, and lifecycle review.

FAQ

Is a dataset manifest the same as a data card?

No. A data card explains context and limitations for people; a manifest binds machine-verifiable files, versions, rules, reports, and hashes. Use both together.

Do checksums prove dataset quality?

No. They prove that bytes match a recorded artifact. Rights, correctness, representation, labels, privacy, and fitness still need separate review.

When should a dataset get a new version?

Issue a new immutable version when records, labels, sources, transformations, splits, schema, permissions, or material documentation change.