Model Cards
A standard document describing what a model is, what it's for, and where it fails — a good idea, universally endorsed, and thinnest exactly where it matters most.
When not to use it
- As evidence of safety. It's a self-report. It tells you what the vendor chose to say.
- Instead of your own evaluation. Their subgroups aren't yours; their conditions aren't your conditions.
- As a compliance box. A card with a vacuous limitations section costs credibility rather than buying it.
- Expecting training data disclosure. On frontier models, it's mostly gone.
Reach for something else instead
- Datasheets for Datasets — for the data, which outlives the model.
- Your own disaggregated evaluation — on your population. The only one that's about you.
- Third-party audits — independent, rare, and the only non-self-reported option.
- Transparency indices — measure what vendors actually disclose rather than what they claim.
Sources & further reading
- Mitchell et al. (2019), Model Cards for Model Reporting — the proposal; disaggregated evaluation is the substance.
- Gebru et al. (2018), Datasheets for Datasets — the same for data, and the dataset outlives the model.
- Bommasani et al. (2023), The Foundation Model Transparency Index — measuring what's actually disclosed. The results are the argument.
Primary sources, listed so you can check the claims on this page rather than take them on trust.
Where people go wrong
- Reading a card as verification. It's a self-report with no auditor.
- Writing a limitations section that says "may sometimes be incorrect." That's worse than nothing.
- Reporting aggregate metrics only — which is the exact thing the proposal exists to fix.
- Assuming a card covers your use case. Intended use is theirs, not yours.
At a glance
FieldSafety & Ethics
What it isstandard documentation: use, data, evaluation, limitations
The substancedisaggregated evaluation; aggregates hide the failures that matter
What's disappearingtraining data disclosure
Statusendorsed everywhere, hollowed out where it counts
DifficultyBeginner
Flashcards for this concept
Question
—
Answer
—
1 / 4
Often compared with
Model card vs. your own evaluation — one describes the model on the vendor's chosen population; the other tells you what happens on yours.