# Augmented Datasheets for Speech Datasets and Ethical Decision-Making

arXiv: https://arxiv.org/abs/2305.04672 · HTML: https://mlsanity.com/paper/2305.04672/

## Key facts

- Cited by 16 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 8 May 2023.

## Built on by (most cited first, 16 of 16)

- [Careless Whisper: Speech-to-Text Hallucination Harms](https://mlsanity.com/paper/2402.08021/) (2024)
- [Navigating Dataset Documentations in AI: A Large-Scale Analysis of Dataset Cards on Hugging Face](https://mlsanity.com/paper/2401.13822/) (2024)
- [Value Imprint: A Technique for Auditing the Human Values Embedded in RLHF Datasets](https://mlsanity.com/paper/2411.11937/) (2024)
- [Lazy Data Practices Harm Fairness Research](https://mlsanity.com/paper/2404.17293/) (2024)
- [Promoting the Responsible Development of Speech Datasets for Mental Health and Neurological Disorders Research](https://mlsanity.com/paper/2406.04116/) (2024)
- [Completeness of Datasets Documentation on ML/AI repositories: an Empirical Investigation](https://mlsanity.com/paper/2503.13463/) (2025)
- [Addressing Pitfalls in Auditing Practices of Automatic Speech Recognition Technologies: A Case Study of People with Aphasia](https://mlsanity.com/paper/2506.08846/) (2025)
- [A Taxonomy of Challenges to Curating Fair Datasets](https://mlsanity.com/paper/2406.06407/) (2024)
- [The Consensus Trap: Dissecting Subjectivity and the "Ground Truth" Illusion in Data Annotation](https://mlsanity.com/paper/2602.11318/) (2026)
- [From Model Performance to Claim: How a Change of Focus in Machine Learning Replicability Can Help Bridge the Responsibility Gap](https://mlsanity.com/paper/2404.13131/) (2024)
- [From Reflection to Repair: A Scoping Review of Dataset Documentation Tools](https://mlsanity.com/paper/2602.15968/) (2026)
- [Fairness-in-the-Workflow: How Machine Learning Practitioners at Big Tech Companies Approach Fairness in Recommender Systems](https://mlsanity.com/paper/2505.19441/) (2025)
- [Large Language Models in Cybersecurity: Applications, Vulnerabilities, and Defense Techniques](https://mlsanity.com/paper/2507.13629/) (2025)
- [Evaluation of AI Ethics Tools in Language Models: A Developers' Perspective Case Study](https://mlsanity.com/paper/2512.15791/) (2025)
- [Queer inclusion in speech datasets: An audit and taxonomy of practical tensions](https://mlsanity.com/paper/2609.25491/) (2026)
- [Towards participatory speech dataset curation: A queer case study and conceptual framework](https://mlsanity.com/paper/2609.25496/) (2026)

## Abstract (arXiv)

Speech datasets are crucial for training Speech Language Technologies (SLT); however, the lack of diversity of the underlying training data can lead to serious limitations in building equitable and robust SLT products, especially along dimensions of language, accent, dialect, variety, and speech impairment - and the intersectionality of speech features with socioeconomic and demographic features. Furthermore, there is often a lack of oversight on the underlying training data - commonly built on massive web-crawling and/or publicly available speech - with regard to the ethics of such data collection. To encourage standardized documentation of such speech data components, we introduce an augmented datasheet for speech datasets, which can be used in addition to "Datasheets for Datasets". We then exemplify the importance of each question in our augmented datasheet based on in-depth literature reviews of speech data used in domains such as machine learning, linguistics, and health. Finally, we encourage practitioners - ranging from dataset creators to researchers - to use our augmented datasheet to better define the scope, properties, and limits of speech datasets, while also encouraging consideration of data-subject protection and user community empowerment. Ethical dataset creation is not a one-size-fits-all process, but dataset creators can use our augmented datasheet to reflexively consider the social context of related SLT applications and data sources in order to foster more inclusive SLT products downstream.