# "Garbage In, Garbage Out" Revisited: What Do Machine Learning Application Papers Report About Human-Labeled Training Data?

arXiv: https://arxiv.org/abs/2107.02278 · HTML: https://mlsanity.com/paper/2107.02278/

## Key facts

- Cited by 24 papers in the map (data through 14 Oct 2026).
- Builds on 19 of its 148 references that are in the map.
- First posted on arXiv 5 Jul 2021.

## Builds on (most cited first, 19 of 19)

- [Equality of Opportunity in Supervised Learning](https://mlsanity.com/paper/1610.02413/) (2016)
- [A Survey on Bias and Fairness in Machine Learning](https://mlsanity.com/paper/1908.09635/) (2019)
- [Datasheets for Datasets](https://mlsanity.com/paper/1803.09010/) (2018)
- [Model Cards for Model Reporting](https://mlsanity.com/paper/1810.03993/) (2018)
- [Language (Technology) is Power: A Critical Survey of "Bias" in NLP](https://mlsanity.com/paper/2005.14050/) (2020)
- [Measurement and Fairness](https://mlsanity.com/paper/1912.05511/) (2019)
- [Large image datasets: A pyrrhic win for computer vision?](https://mlsanity.com/paper/2006.16923/) (2020)
- [A comparative study of fairness-enhancing interventions in machine learning](https://mlsanity.com/paper/1802.04422/) (2018)
- [End-to-End Bias Mitigation by Modelling Biases in Corpora](https://mlsanity.com/paper/1909.06321/) (2019)
- [Towards a Critical Race Methodology in Algorithmic Fairness](https://mlsanity.com/paper/1912.03593/) (2019)
- [The Dataset Nutrition Label: A Framework To Drive Higher Data Quality Standards](https://mlsanity.com/paper/1805.03677/) (2018)
- [Garbage In, Garbage Out? Do Machine Learning Application Papers in Social Computing Report Where Human-Labeled Training Data Comes From?](https://mlsanity.com/paper/1912.08320/) (2019)
- [FactSheets: Increasing Trust in AI Services through Supplier's Declarations of Conformity](https://mlsanity.com/paper/1808.07261/) (2018)
- [Multilingual Twitter Sentiment Classification: The Role of Human Annotators](https://mlsanity.com/paper/1602.07563/) (2016)
- [Responses to Critiques on Machine Learning of Criminality Perceptions (Addendum of arXiv:1611.04135)](https://mlsanity.com/paper/1611.04135/) (2016)
- [ABOUT ML: Annotation and Benchmarking on Understanding and Transparency of Machine Learning Lifecycles](https://mlsanity.com/paper/1912.06166/) (2019)
- [Towards Traceability in Data Ecosystems using a Bill of Materials Model](https://mlsanity.com/paper/1904.04253/) (2019)
- [Good Enough Practices in Scientific Computing](https://mlsanity.com/paper/1609.00037/) (2016)
- [10 Simple Rules for the Care and Feeding of Scientific Data](https://mlsanity.com/paper/1401.2134/) (2014)

## Built on by (most cited first, 24 of 24)

- [Measuring what Matters: Construct Validity in Large Language Model Benchmarks](https://mlsanity.com/paper/2511.04703/) (2025)
- [Ground-Truth, Whose Truth? -- Examining the Challenges with Annotating Toxic Text Datasets](https://mlsanity.com/paper/2112.03529/) (2021)
- [Ground Truth Or Dare: Factors Affecting The Creation Of Medical Datasets For Training AI](https://mlsanity.com/paper/2309.12327/) (2023)
- [Reproducibility in Multiple Instance Learning: A Case For Algorithmic Unit Tests](https://mlsanity.com/paper/2310.17867/) (2023)
- [Closing the Loop: Testing ChatGPT to Generate Model Explanations to Improve Human Labelling of Sponsored Content on Social Media](https://mlsanity.com/paper/2306.05115/) (2023)
- [A Siren Song of Open Source Reproducibility](https://mlsanity.com/paper/2204.04372/) (2022)
- [Envisioning Narrative Intelligence: A Creative Visual Storytelling Anthology](https://mlsanity.com/paper/2310.04529/) (2023)
- [Rigor in AI: Doing Rigorous AI Work Requires a Broader, Responsible AI-Informed Conception of Rigor](https://mlsanity.com/paper/2506.14652/) (2025)
- [The Unseen Targets of Hate -- A Systematic Review of Hateful Communication Datasets](https://mlsanity.com/paper/2405.08562/) (2024)
- [Mix-Pooling Strategy for Attention Mechanism](https://mlsanity.com/paper/2208.10322/) (2022)
- [Toxic language detection: a systematic review of Arabic datasets](https://mlsanity.com/paper/2312.07228/) (2023)
- [MEGAnno: Exploratory Labeling for NLP in Computational Notebooks](https://mlsanity.com/paper/2301.03095/) (2023)
- [The Epistemological Consequences of Large Language Models: Rethinking collective intelligence and institutional knowledge](https://mlsanity.com/paper/2512.19570/) (2025)
- [From Ground Truth to Measurement: A Statistical Framework for Human Labeling](https://mlsanity.com/paper/2604.07591/) (2026)
- [Changing Data Sources in the Age of Machine Learning for Official Statistics](https://mlsanity.com/paper/2306.04338/) (2023)
- [Personalization of Stress Mobile Sensing using Self-Supervised Learning](https://mlsanity.com/paper/2308.02731/) (2023)
- [Best practices for machine learning in antibody discovery and development](https://mlsanity.com/paper/2312.08470/) (2023)
- [Situated Ground Truths: Enhancing Bias-Aware AI by Situating Data Labels with SituAnnotate](https://mlsanity.com/paper/2406.07583/) (2024)
- [A quest through interconnected datasets: lessons from highly-cited ICASSP papers](https://mlsanity.com/paper/2410.03676/) (2024)
- [Shape Modeling of Longitudinal Medical Images: From Diffeomorphic Metric Mapping to Deep Learning](https://mlsanity.com/paper/2503.21489/) (2025)
- [Measuring Corporate Human Capital Disclosures: Lexicon, Data, Code, and Research Opportunities](https://mlsanity.com/paper/2506.10155/) (2025)
- [RangeAD: Fast On-Model Anomaly Detection](https://mlsanity.com/paper/2603.17795/) (2026)
- [Architecture-agnostic Lipschitz-constant Bayesian header and its application to resolve semantically proximal classification errors with vision transformers](https://mlsanity.com/paper/2605.05908/) (2026)
- [Algebraic Multigrid Acceleration for Efficient Label Spreading](https://mlsanity.com/paper/2608.26309/) (2026)

## Abstract (arXiv)

Supervised machine learning, in which models are automatically derived from labeled training data, is only as good as the quality of that data. This study builds on prior work that investigated to what extent 'best practices' around labeling training data were followed in applied ML publications within a single domain (social media platforms). In this paper, we expand by studying publications that apply supervised ML in a far broader spectrum of disciplines, focusing on human-labeled data. We report to what extent a random sample of ML application papers across disciplines give specific details about whether best practices were followed, while acknowledging that a greater range of application fields necessarily produces greater diversity of labeling and annotation methods. Because much of machine learning research and education only focuses on what is done once a "ground truth" or "gold standard" of training data is available, it is especially relevant to discuss issues around the equally-important aspect of whether such data is reliable in the first place. This determination becomes increasingly complex when applied to a variety of specialized fields, as labeling can range from a task requiring little-to-no background knowledge to one that must be performed by someone with career expertise.