Model Card for AI Projects: Scope, Data, Metrics, Limits, Monitoring

An AI model card is a concise, versioned document that explains what a model is, its intended and excluded uses, the evidence used to evaluate it, important limitations, responsible owners, and the conditions for deployment and monitoring. It helps a reader decide whether the model fits a specific context; it is not a certificate of fairness, safety, accuracy, or legal compliance.

Learn with a bounded, low-stakes project and synthetic or properly governed data. High-impact uses involving health, employment, education access, credit, law, safety, or essential services need domain specialists, applicable governance, independent evaluation, human oversight, security, privacy review, and recourse beyond this beginner guide.

Identify The Exact Model

Start with a stable model name, version, release date, artifact or endpoint identifier, owner, contact route, license or access terms, and current lifecycle state. Include the preprocessing, tokenizer, feature pipeline, prompt template, retrieval configuration, or threshold version needed to reproduce actual behavior.

Distinguish the base model from the deployed system. An application may add retrieval, rules, tools, moderation, caching, and human review that materially change outcomes. Link the card to one immutable release package rather than describing a product name that changes silently.

Name The Card Audience

State who should use the card: developers integrating an artifact, reviewers approving a release, operators monitoring it, domain experts assessing suitability, or affected users seeking an explanation. Define the decisions each audience can make from the document.

Use plain language before specialist metrics. Google Research presents model cards as context for downstream users, including intended uses and limitations. A card that only its original model authors understand cannot support accountable reuse.

Describe The Task

Explain the input, output, unit of prediction or generation, supported languages or modalities, operating environment, and action that follows an output. Give representative examples using fabricated data. State whether scores are rankings, similarities, probabilities, labels, generated text, or another construct.

Avoid marketing labels such as intelligent, unbiased, human-like, or enterprise-ready. Describe observable behavior and tested conditions. If the model depends on a fixed taxonomy, schema, knowledge cutoff, or external data source, make that boundary visible.

Define Intended Uses

List specific supported workflows, users, data conditions, and decision roles. Include the expected level of human review and whether outputs inform, recommend, draft, prioritize, or execute. Tie each use to evaluation evidence and operational controls.

A broad statement such as for education is insufficient. A low-stakes practice classifier for synthetic course feedback differs from a system affecting admission or grading. Describe the real consequence, scale, and accountable operator.

State Excluded Uses

Name contexts that were not evaluated, conflict with the license or policy, exceed the taxonomy, require unsupported languages, involve prohibited content, or create unacceptable consequences. Explain what a user should do instead, such as obtain expert review or use a validated domain system.

Excluded use is not a disclaimer that transfers all responsibility to users. Add technical and organizational controls where misuse is foreseeable: access restrictions, safe defaults, review requirements, rate limits, logging, and monitoring.

Link The Data Manifest

Summarize training, tuning, calibration, and evaluation datasets with sources, time range, geography, languages, collection purpose, consent or permissions, labels, exclusions, transformations, and known gaps. Link immutable dataset versions rather than embedding an untraceable description.

Use a dataset version manifest to record sources, splits, rights, and hashes. Do not expose personal records, confidential examples, security-sensitive data, or restricted locations in the public card.

Explain Label And Review Policy

For supervised tasks, define each label, edge cases, precedence rules, annotator qualifications, disagreement handling, adjudication, and measured agreement. For generative evaluation, document rubrics, reviewer instructions, sample selection, and how subjective judgments were combined.

Labels are claims about data, not ground truth by default. Record uncertainty, missingness, and disagreement. A high metric calculated from inconsistent labels can make a weak system appear precise.

Document The Evaluation Design

Describe test splits, collection dates, sample sizes, inclusion rules, baseline systems, test environment, random seeds where relevant, and measures chosen before final evaluation. Explain why the evaluation represents intended use and where it does not.

Keep threshold selection and final testing separate. Prevent leakage from duplicated records, users, time periods, or derived examples. If the production distribution differs from the test set, state the expected shift rather than presenting one score as universal.

Report Metrics With Context

Provide metrics that match the task and consequences: precision and recall by class, calibration, ranking measures, error rates, abstention, latency, robustness, or human-rated dimensions. Include denominators, uncertainty intervals where supported, threshold values, and baseline comparisons.

Do not lead with accuracy when class imbalance or asymmetric harm makes it misleading. Explain false positives, false negatives, and rejected cases in operational terms. Report what happens after model output, not only isolated benchmark performance.

Evaluate Relevant Groups

Google’s model-card research recommends performance reporting across cultural, demographic, phenotypic, and intersectional groups relevant to intended domains. Select groups from context and potential impact, not from whatever columns happen to be available. Involve domain and affected-community expertise where appropriate.

Report sample sizes and uncertainty, protect privacy, and avoid publishing unstable conclusions about very small groups. Similar aggregate performance does not establish fairness, while observed differences require investigation of data, labels, deployment, and consequences rather than simplistic ranking.

Catalog Failure Modes

List known errors with realistic examples: ambiguous input, missing context, unsupported language, distribution shift, adversarial manipulation, hallucinated content, confident mistakes, degraded images, long inputs, or upstream outages. Explain detection and containment for each important failure.

Include failures discovered after release and link incidents without exposing exploitable details. A model card should become more useful as evidence accumulates. Removing an embarrassing limitation makes downstream harm more likely and undermines the document’s purpose.

Describe Uncertainty And Abstention

Explain what confidence-like scores represent, how they were calibrated, which thresholds govern automation, and when the system declines to decide. State review queue ownership, capacity assumptions, timeout behavior, and fallback routes.

Use an explicit abstention and review workflow for uncertain, out-of-scope, policy-restricted, or malformed cases. Do not interpret a model’s fluent explanation as reliable uncertainty evidence without validation.

Record Safety And Privacy Controls

Summarize data minimization, access control, retention, encryption, abuse testing, content safeguards, human approval, audit logging, incident response, and appeal mechanisms applicable to the release. Name owners and link controlled internal evidence where details cannot be public.

State residual risk honestly. A filter may reduce tested failures without eliminating them, and monitoring cannot recover information already disclosed. Do not include secrets, private prompts, personal examples, system vulnerabilities, or bypass instructions in a public card.

Explain Deployment Conditions

Document supported hardware or service, dependencies, regions, latency targets, throughput, cost boundaries, failover, rate limits, and required upstream data quality. State whether the model is advisory, shadowed, gradually rolled out, or permitted to trigger automatic actions.

Identify version compatibility across model, code, features, prompts, schemas, and policies. A model evaluated in a notebook has not automatically been evaluated behind a changing API, retrieval index, or user interface.

Assign Human Oversight

Name who reviews outputs, which cases require expertise, what evidence reviewers see, how automation bias is addressed, and how corrections, overrides, appeals, and escalations are recorded. Define who can pause the system and who approves threshold or policy changes.

Human in the loop is not a complete control when reviewers lack time, authority, context, or feedback. Measure queue delay, agreement, overrides, sampled automated errors, and final outcomes. Protect reviewers from unnecessary sensitive data.

Define Monitoring

Set a dated baseline and track input drift, missing fields, score distributions, output mix, errors from labelled audits, subgroup behavior, abstention, overrides, complaints, incidents, latency, cost, and upstream failures. Assign thresholds, investigation owners, and response deadlines.

NIST’s AI RMF organizes ongoing work around governance, mapping, measurement, and management. Monitoring should lead to defined actions such as investigation, rollback, restricted use, new evaluation, data repair, or card revision rather than an unattended dashboard.

Version Every Material Change

Update the card when the model, data, labels, preprocessing, prompts, retrieval, thresholds, policy, interface, deployment, or intended use changes materially. Preserve a changelog, previous versions, approval, evidence hashes, and effective date.

Do not edit a card in place so that it describes evidence from a different release. Link each public statement to the tested package. Deprecate old versions clearly and specify migration or rollback behavior for integrators.

Review And Publish

Have model, data, domain, product, privacy, security, accessibility, operations, and governance owners review applicable sections. Resolve contradictions between intended use, metrics, limitations, and deployment controls. Publish at a stable URL accessible to the people who rely on the model.

The Artificial Intelligence course can build foundations in data, models, evaluation, and deployment. Maintain the card after launch, because transparent documentation is a lifecycle practice rather than a one-time release note.

FAQ

Is a model card a safety certification?

No. It reports context, evidence, limits, and controls so readers can judge suitability; independent assurance may still be required.

Does every AI system need the same card?

No. Sections and depth should match the model, audience, intended use, impact, and available evidence.

When should a model card change?

Update it whenever a material model, data, policy, evaluation, deployment, or intended-use change makes the existing evidence incomplete.