Documenting models and data
Model cards and datasheets: the two documents to request before deploying a system.
ObjectiveKnow what a model card and a datasheet for a dataset contain, and what to look for in them.
In 2019, researchers proposed model cards. These are short documents that describe a model’s intended use, its performance measured by group, its known limitations and the uses it is not recommended for. Datasheets for datasets do the same for a dataset: provenance, composition, collection, consent and updates.
What to look for
- The intended use, and above all the uses for which the model has not been evaluated.
- Results broken down by group, not just an average.
- The date and provenance of the data.
- The limitations the vendor itself acknowledges.
References
- Mitchell, Wu, Zaldivar et al. (2019). Model Cards for Model Reporting. Proceedings of FAT* 2019, p. 220–229. doi.org/10.1145/3287560.3287596
- Gebru, Morgenstern, Vecchione et al. (2021). Datasheets for Datasets. Communications of the ACM 64(12), p. 86–92. doi.org/10.1145/3458723