# Evaluation Gaps in Machine Learning Practice

arXiv: https://arxiv.org/abs/2205.05256 · HTML: https://mlsanity.com/paper/2205.05256/

## Key facts

- Cited by 49 papers in the map (data through 14 Oct 2026).
- Builds on 83 of its 188 references that are in the map.
- First posted on arXiv 11 May 2022.

## Builds on (most cited first, 30 of 83)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks](https://mlsanity.com/paper/1506.01497/) (2015)
- [GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding](https://mlsanity.com/paper/1804.07461/) (2018)
- [On the Opportunities and Risks of Foundation Models](https://mlsanity.com/paper/2108.07258/) (2021)
- [Benchmarking Neural Network Robustness to Common Corruptions and Perturbations](https://mlsanity.com/paper/1903.12261/) (2019)
- [Extracting Training Data from Large Language Models](https://mlsanity.com/paper/2012.07805/) (2020)
- [Equality of Opportunity in Supervised Learning](https://mlsanity.com/paper/1610.02413/) (2016)
- [Concrete Problems in AI Safety](https://mlsanity.com/paper/1606.06565/) (2016)
- [Datasheets for Datasets](https://mlsanity.com/paper/1803.09010/) (2018)
- [WILDS: A Benchmark of in-the-Wild Distribution Shifts](https://mlsanity.com/paper/2012.07421/) (2020)
- [Energy and Policy Considerations for Deep Learning in NLP](https://mlsanity.com/paper/1906.02243/) (2019)
- [Model Cards for Model Reporting](https://mlsanity.com/paper/1810.03993/) (2018)
- [Beyond Accuracy: Behavioral Testing of NLP models with CheckList](https://mlsanity.com/paper/2005.04118/) (2020)
- [Generalizing to Unseen Domains: A Survey on Domain Generalization](https://mlsanity.com/paper/2103.03097/) (2021)
- [Fair prediction with disparate impact: A study of bias in recidivism prediction instruments](https://mlsanity.com/paper/1610.07524/) (2016)
- [Gender Bias in Coreference Resolution: Evaluation and Debiasing Methods](https://mlsanity.com/paper/1804.06876/) (2018)
- [Green AI](https://mlsanity.com/paper/1907.10597/) (2019)
- [Underspecification Presents Challenges for Credibility in Modern Machine Learning](https://mlsanity.com/paper/2011.03395/) (2020)
- [Nuanced Metrics for Measuring Unintended Bias with Real Data for Text Classification](https://mlsanity.com/paper/1903.04561/) (2019)
- [Algorithmic decision making and the cost of fairness](https://mlsanity.com/paper/1701.08230/) (2017)
- [Closing the AI Accountability Gap: Defining an End-to-End Framework for Internal Algorithmic Auditing](https://mlsanity.com/paper/2001.00973/) (2020)
- [AI and the Everything in the Whole Wide World Benchmark](https://mlsanity.com/paper/2111.15366/) (2021)
- [Dealing with Disagreements: Looking Beyond the Majority Vote in Subjective Annotations](https://mlsanity.com/paper/2110.05719/) (2021)
- [Measurement and Fairness](https://mlsanity.com/paper/1912.05511/) (2019)
- [Improving fairness in machine learning systems: What do industry practitioners need?](https://mlsanity.com/paper/1812.05239/) (2018)
- [Towards the Systematic Reporting of the Energy and Carbon Footprints of Machine Learning](https://mlsanity.com/paper/2002.05651/) (2020)
- [Machine Learning Testing: Survey, Landscapes and Horizons](https://mlsanity.com/paper/1906.10742/) (2019)
- [What Will it Take to Fix Benchmarking in Natural Language Understanding?](https://mlsanity.com/paper/2104.02145/) (2021)
- [Fairness in Machine Learning](https://mlsanity.com/paper/2012.15816/) (2020)
- [Contrastive Training for Improved Out-of-Distribution Detection](https://mlsanity.com/paper/2007.05566/) (2020)

## Built on by (most cited first, 30 of 49)

- [Measuring what Matters: Construct Validity in Large Language Model Benchmarks](https://mlsanity.com/paper/2511.04703/) (2025)
- [A Sober Look at Progress in Language Model Reasoning: Pitfalls and Paths to Reproducibility](https://mlsanity.com/paper/2504.07086/) (2025)
- [Evaluating the Social Impact of Generative AI Systems in Systems and Society](https://mlsanity.com/paper/2306.05949/) (2023)
- [Can We Trust AI Benchmarks? An Interdisciplinary Review of Current Issues in AI Evaluation](https://mlsanity.com/paper/2502.06559/) (2025)
- [From plane crashes to algorithmic harm: applicability of safety engineering frameworks for responsible ML](https://mlsanity.com/paper/2210.03535/) (2022)
- [Evaluate & Evaluation on the Hub: Better Best Practices for Data and Model Measurements](https://mlsanity.com/paper/2210.01970/) (2022)
- [Cultural Incongruencies in Artificial Intelligence](https://mlsanity.com/paper/2211.13069/) (2022)
- [Towards Geographic Inclusion in the Evaluation of Text-to-Image Models](https://mlsanity.com/paper/2405.04457/) (2024)
- [Investigating Practices and Opportunities for Cross-functional Collaboration around AI Fairness in Industry Practice](https://mlsanity.com/paper/2306.06542/) (2023)
- ["It's how you do things that matters": Attending to Process to Better Serve Indigenous Communities with Language Technologies](https://mlsanity.com/paper/2402.02639/) (2024)
- [Paradigms of AI Evaluation: Mapping Goals, Methodologies and Culture](https://mlsanity.com/paper/2502.15620/) (2025)
- [Making Intelligence: Ethical Values in IQ and ML Benchmarks](https://mlsanity.com/paper/2209.00692/) (2022)
- [Deep Learning Safety Concerns in Automated Driving Perception](https://mlsanity.com/paper/2309.03774/) (2023)
- [Efficient Lifelong Model Evaluation in an Era of Rapid Progress](https://mlsanity.com/paper/2402.19472/) (2024)
- [AI Mismatches: Identifying Potential Algorithmic Harms Before AI Development](https://mlsanity.com/paper/2502.18682/) (2025)
- [Rigor in AI: Doing Rigorous AI Work Requires a Broader, Responsible AI-Informed Conception of Rigor](https://mlsanity.com/paper/2506.14652/) (2025)
- [Challenges to Evaluating the Generalization of Coreference Resolution Models: A Measurement Modeling Perspective](https://mlsanity.com/paper/2303.09092/) (2023)
- [Improving the Validity and Practical Usefulness of AI/ML Evaluations Using an Estimands Framework](https://mlsanity.com/paper/2406.10366/) (2024)
- [Adding Chocolate to Mint: Mitigating Metric Interference in Machine Translation](https://mlsanity.com/paper/2503.08327/) (2025)
- [Position: Causal Machine Learning Requires Rigorous Synthetic Experiments for Broader Adoption](https://mlsanity.com/paper/2508.08883/) (2025)
- [Measuring What Matters: Connecting AI Ethics Evaluations to System Attributes, Hazards, and Harms](https://mlsanity.com/paper/2510.10339/) (2025)
- [Quantitative AI Risk Assessments: Opportunities and Challenges](https://mlsanity.com/paper/2209.06317/) (2022)
- [Test & Evaluation Best Practices for Machine Learning-Enabled Systems](https://mlsanity.com/paper/2310.06800/) (2023)
- [(Beyond) Reasonable Doubt: Challenges that Public Defenders Face in Scrutinizing AI in Court](https://mlsanity.com/paper/2403.13004/) (2024)
- [Designing Speech Technologies for Australian Aboriginal English: Opportunities, Risks and Participation](https://mlsanity.com/paper/2503.03186/) (2025)
- [The Double Measurement Confound in Agent Benchmarks: De-Scaffolding, Ground-Truth Scoring, and Reliability Beyond the Mean](https://mlsanity.com/paper/2609.09218/) (2026)
- [A Conceptual Framework for Ethical Evaluation of Machine Learning Systems](https://mlsanity.com/paper/2408.10239/) (2024)
- [Rethinking Optimization: A Systems-Based Approach to Social Externalities](https://mlsanity.com/paper/2506.12825/) (2025)
- [Knowledge without Wisdom: Measuring Misalignment between LLMs and Intended Impact](https://mlsanity.com/paper/2603.00883/) (2026)
- [A Unified Framework to Quantify Cultural Intelligence of AI](https://mlsanity.com/paper/2603.01211/) (2026)

## Abstract (arXiv)

Forming a reliable judgement of a machine learning (ML) model's appropriateness for an application ecosystem is critical for its responsible use, and requires considering a broad range of factors including harms, benefits, and responsibilities. In practice, however, evaluations of ML models frequently focus on only a narrow range of decontextualized predictive behaviours. We examine the evaluation gaps between the idealized breadth of evaluation concerns and the observed narrow focus of actual evaluations. Through an empirical study of papers from recent high-profile conferences in the Computer Vision and Natural Language Processing communities, we demonstrate a general focus on a handful of evaluation methods. By considering the metrics and test data distributions used in these methods, we draw attention to which properties of models are centered in the field, revealing the properties that are frequently neglected or sidelined during evaluation. By studying these properties, we demonstrate the machine learning discipline's implicit assumption of a range of commitments which have normative impacts; these include commitments to consequentialism, abstractability from context, the quantifiability of impacts, the limited role of model inputs in evaluation, and the equivalence of different failure modes. Shedding light on these assumptions enables us to question their appropriateness for ML system contexts, pointing the way towards more contextualized evaluation methodologies for robustly examining the trustworthiness of ML models