# Reality Check: A New Evaluation Ecosystem Is Necessary to Understand AI's Real World Effects

arXiv: https://arxiv.org/abs/2505.18893 · HTML: https://mlsanity.com/paper/2505.18893/

## Key facts

- Cited by 19 papers in the map (data through 14 Oct 2026).
- Builds on 54 of its 104 references that are in the map.
- First posted on arXiv 24 May 2025.

## Builds on (most cited first, 30 of 54)

- [Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena](https://mlsanity.com/paper/2306.05685/) (2023)
- [Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learning](https://mlsanity.com/paper/1506.02142/) (2015)
- [Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback](https://mlsanity.com/paper/2204.05862/) (2022)
- [Extracting Training Data from Large Language Models](https://mlsanity.com/paper/2012.07805/) (2020)
- [Survey of Hallucination in Natural Language Generation](https://mlsanity.com/paper/2202.03629/) (2022)
- [Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language models](https://mlsanity.com/paper/2206.04615/) (2022)
- [Do ImageNet Classifiers Generalize to ImageNet?](https://mlsanity.com/paper/1902.10811/) (2019)
- [Not what you've signed up for: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injection](https://mlsanity.com/paper/2302.12173/) (2023)
- [Jailbreaking Black Box Large Language Models in Twenty Queries](https://mlsanity.com/paper/2310.08419/) (2023)
- [RealToxicityPrompts: Evaluating Neural Toxic Degeneration in Language Models](https://mlsanity.com/paper/2009.11462/) (2020)
- [A Survey on Evaluation of Large Language Models](https://mlsanity.com/paper/2307.03109/) (2023)
- [Open Problems and Fundamental Limitations of Reinforcement Learning from Human Feedback](https://mlsanity.com/paper/2307.15217/) (2023)
- [Quantifying Memorization Across Neural Language Models](https://mlsanity.com/paper/2202.07646/) (2022)
- [The WMDP Benchmark: Measuring and Reducing Malicious Use With Unlearning](https://mlsanity.com/paper/2403.03218/) (2024)
- [DecodingTrust: A Comprehensive Assessment of Trustworthiness in GPT Models](https://mlsanity.com/paper/2306.11698/) (2023)
- [Underspecification Presents Challenges for Credibility in Modern Machine Learning](https://mlsanity.com/paper/2011.03395/) (2020)
- [Dynabench: Rethinking Benchmarking in NLP](https://mlsanity.com/paper/2104.14337/) (2021)
- [Rainbow Teaming: Open-Ended Generation of Diverse Adversarial Prompts](https://mlsanity.com/paper/2402.16822/) (2024)
- [Do Membership Inference Attacks Work on Large Language Models?](https://mlsanity.com/paper/2402.07841/) (2024)
- [Beyond Memorization: Violating Privacy Via Inference with Large Language Models](https://mlsanity.com/paper/2310.07298/) (2023)
- [Show Your Work: Improved Reporting of Experimental Results](https://mlsanity.com/paper/1909.03004/) (2019)
- [Stealing Part of a Production Language Model](https://mlsanity.com/paper/2403.06634/) (2024)
- [Sociotechnical Safety Evaluation of Generative AI Systems](https://mlsanity.com/paper/2310.11986/) (2023)
- [LLM Defenses Are Not Robust to Multi-Turn Human Jailbreaks Yet](https://mlsanity.com/paper/2408.15221/) (2024)
- [TaskBench: Benchmarking Large Language Models for Task Automation](https://mlsanity.com/paper/2311.18760/) (2023)
- [Power to the People? Opportunities and Challenges for Participatory AI](https://mlsanity.com/paper/2209.07572/) (2022)
- [Sociotechnical Harms of Algorithmic Systems: Scoping a Taxonomy for Harm Reduction](https://mlsanity.com/paper/2210.05791/) (2022)
- [Sponge Examples: Energy-Latency Attacks on Neural Networks](https://mlsanity.com/paper/2006.03463/) (2020)
- [Participation is not a Design Fix for Machine Learning](https://mlsanity.com/paper/2007.02423/) (2020)
- [Which Economic Tasks are Performed with AI? Evidence from Millions of Claude Conversations](https://mlsanity.com/paper/2503.04761/) (2025)

## Built on by (most cited first, 19 of 19)

- [OneMillion-Bench: How Far are Language Agents from Human Experts?](https://mlsanity.com/paper/2603.07980/) (2026)
- [VERA-MH Concept Paper](https://mlsanity.com/paper/2510.15297/) (2025)
- [Evaluation Cards: An Interpretive Layer for AI Evaluation Reporting](https://mlsanity.com/paper/2606.09809/) (2026)
- [Red Teaming AI Red Teaming](https://mlsanity.com/paper/2507.05538/) (2025)
- [CIRCLE: A Framework for Evaluating AI from a Real-World Lens](https://mlsanity.com/paper/2602.24055/) (2026)
- [Dimensional Characterization and Pathway Modeling for Catastrophic AI Risks](https://mlsanity.com/paper/2508.06411/) (2025)
- [Ask What Your Country Can Do For You: Towards a Public Red Teaming Model](https://mlsanity.com/paper/2510.20061/) (2025)
- [From Symptoms to Systems: An Expert-Guided Approach to Understanding Risks of Generative AI for Eating Disorders](https://mlsanity.com/paper/2512.04843/) (2025)
- [Making AI Evaluation Deployment Relevant Through Context Specification](https://mlsanity.com/paper/2603.06811/) (2026)
- [ContractScrub: A benchmark for final review of legal contracts](https://mlsanity.com/paper/2608.20204/) (2026)
- [Branching Out: Broadening AI Measurement and Evaluation with Measurement Trees](https://mlsanity.com/paper/2509.26632/) (2025)
- [Towards Real-World Validity in Generative AI Benchmarks: Understanding and Designing Domain-Centered Evaluations for Journalism Practitioners](https://mlsanity.com/paper/2511.05501/) (2025)
- [The AI Consumer Index (ACE)](https://mlsanity.com/paper/2512.04921/) (2025)
- [Red Teaming LLMs as Socio-Technical Practice: From Exploration and Data Creation to Evaluation](https://mlsanity.com/paper/2602.18483/) (2026)
- [RCTs for Frontier AI Governance: Methodological Challenges and Solutions for Human Uplift Studies](https://mlsanity.com/paper/2603.11001/) (2026)
- [Real-World AI Evaluation: How FRAME Generates Systematic Evidence to Resolve the Decision-Maker's Dilemma](https://mlsanity.com/paper/2603.13294/) (2026)
- [The Foreign Policy AI Evaluation Gap](https://mlsanity.com/paper/2607.02955/) (2026)
- [Toward Human Rights Benchmarking for LLMs: A Pilot Methodology](https://mlsanity.com/paper/2608.10268/) (2026)
- [An Investigation of the NeurIPS and ICML 2025 Position Tracks](https://mlsanity.com/paper/2608.16894/) (2026)

## Abstract (arXiv)

Conventional AI evaluation approaches concentrated within the AI stack exhibit systemic limitations for exploring, navigating and resolving the human and societal factors that play out in real world deployment such as in education, finance, healthcare, and employment sectors. AI capability evaluations can capture detail about first-order effects, such as whether immediate system outputs are accurate, or contain toxic, biased or stereotypical content, but AI's second-order effects, i.e. any long-term outcomes and consequences that may result from AI use in the real world, have become a significant area of interest as the technology becomes embedded in our daily lives. These secondary effects can include shifts in user behavior, societal, cultural and economic ramifications, workforce transformations, and long-term downstream impacts that may result from a broad and growing set of risks. This position paper argues that measuring the indirect and secondary effects of AI will require expansion beyond static, single-turn approaches conducted in silico to include testing paradigms that can capture what actually materializes when people use AI technology in context. Specifically, we describe the need for data and methods that can facilitate contextual awareness and enable downstream interpretation and decision making about AI's secondary effects, and recommend requirements for a new ecosystem.