# Evaluating a Methodology for Increasing AI Transparency: A Case Study

arXiv: https://arxiv.org/abs/2201.13224 · HTML: https://mlsanity.com/paper/2201.13224/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 9 of its 28 references that are in the map.
- First posted on arXiv 24 Jan 2022.

## Builds on (most cited first, 9 of 9)

- [Datasheets for Datasets](https://mlsanity.com/paper/1803.09010/) (2018)
- [Model Cards for Model Reporting](https://mlsanity.com/paper/1810.03993/) (2018)
- [How do Data Science Workers Collaborate? Roles, Workflows, and Tools](https://mlsanity.com/paper/2001.06684/) (2020)
- [How AI Developers Overcome Communication Challenges in a Multidisciplinary Team: A Case Study](https://mlsanity.com/paper/2101.06098/) (2021)
- [The Dataset Nutrition Label (2nd Gen): Leveraging Context to Mitigate Harms in Artificial Intelligence](https://mlsanity.com/paper/2201.03954/) (2022)
- [Experiences with Improving the Transparency of AI Models and Services](https://mlsanity.com/paper/1911.08293/) (2019)
- [ABOUT ML: Annotation and Benchmarking on Understanding and Transparency of Machine Learning Lifecycles](https://mlsanity.com/paper/1912.06166/) (2019)
- [Towards evaluating and eliciting high-quality documentation for intelligent systems](https://mlsanity.com/paper/2011.08774/) (2020)
- [Canada Protocol: an ethical checklist for the use of Artificial Intelligence in Suicide Prevention and Mental Health](https://mlsanity.com/paper/1907.07493/) (2019)

## Built on by (most cited first, 3 of 3)

- [Requirements Engineering for Machine Learning: A Review and Reflection](https://mlsanity.com/paper/2210.00859/) (2022)
- [Compliance Cards: Automated EU AI Act Compliance Analyses amidst a Complex AI Supply Chain](https://mlsanity.com/paper/2406.14758/) (2024)
- [Computational Compliance for AI Regulation: Blueprint for a New Research Domain](https://mlsanity.com/paper/2601.04474/) (2026)

## Abstract (arXiv)

In reaction to growing concerns about the potential harms of artificial intelligence (AI), societies have begun to demand more transparency about how AI models and systems are created and used. To address these concerns, several efforts have proposed documentation templates containing questions to be answered by model developers. These templates provide a useful starting point, but no single template can cover the needs of diverse documentation consumers. It is possible in principle, however, to create a repeatable methodology to generate truly useful documentation. Richards et al. [25] proposed such a methodology for identifying specific documentation needs and creating templates to address those needs. Although this is a promising proposal, it has not been evaluated. This paper presents the first evaluation of this user-centered methodology in practice, reporting on the experiences of a team in the domain of AI for healthcare that adopted it to increase transparency for several AI models. The methodology was found to be usable by developers not trained in user-centered techniques, guiding them to creating a documentation template that addressed the specific needs of their consumers while still being reusable across different models and use cases. Analysis of the benefits and costs of this methodology are reviewed and suggestions for further improvement in both the methodology and supporting tools are summarized.