mlsanityData to 7 Oct 2026

Reality Check: A New Evaluation Ecosystem Is Necessary to Understand AI's Real World Effects

arXiv 2505.18893 ↗First posted 24 May 2025cs.CY cs.AIcited by 18

Not on the map. Why?

Abstract (arXiv)

Conventional AI evaluation approaches concentrated within the AI stack exhibit systemic limitations for exploring, navigating and resolving the human and societal factors that play out in real world deployment such as in education, finance, healthcare, and employment sectors. AI capability evaluations can capture detail about first-order effects, such as whether immediate system outputs are accurate, or contain toxic, biased or stereotypical content, but AI's second-order effects, i.e. any long-term outcomes and consequences that may result from AI use in the real world, have become a significant area of interest as the technology becomes embedded in our daily lives. These secondary effects can include shifts in user behavior, societal, cultural and economic ramifications, workforce transformations, and long-term downstream impacts that may result from a broad and growing set of risks. This position paper argues that measuring the indirect and secondary effects of AI will require expansion beyond static, single-turn approaches conducted in silico to include testing paradigms that can capture what actually materializes when people use AI technology in context. Specifically, we describe the need for data and methods that can facilitate contextual awareness and enable downstream interpretation and decision making about AI's secondary effects, and recommend requirements for a new ecosystem.

Key facts. Cited by 18 papers in the map (data through 7 Oct 2026). Builds on 54 of its 104 references that are in the map. First posted on arXiv 24 May 2025.

Builds on

54 of 104 references.

References · 54

  1. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  2. 2015Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learningcited by 4,416
  3. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,634
  4. 2020Extracting Training Data from Large Language Modelscited by 2,144
  5. 2022Survey of Hallucination in Natural Language Generationcited by 2,056
  6. 2022Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language modelscited by 1,970
  7. 2019Do ImageNet Classifiers Generalize to ImageNet?cited by 1,507
  8. 2023Not what you've signed up for: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injectioncited by 1,339
  9. 2023Jailbreaking Black Box Large Language Models in Twenty Queriescited by 1,326
  10. 2020RealToxicityPrompts: Evaluating Neural Toxic Degeneration in Language Modelscited by 1,235
  11. 2023A Survey on Evaluation of Large Language Modelscited by 1,227
  12. 2022Quantifying Memorization Across Neural Language Modelscited by 755
38 more
  1. 2023Open Problems and Fundamental Limitations of Reinforcement Learning from Human Feedbackcited by 743
  2. 2023DecodingTrust: A Comprehensive Assessment of Trustworthiness in GPT Modelscited by 505
  3. 2020Underspecification Presents Challenges for Credibility in Modern Machine Learningcited by 488
  4. 2024The WMDP Benchmark: Measuring and Reducing Malicious Use With Unlearningcited by 488
  5. 2021Dynabench: Rethinking Benchmarking in NLPcited by 447
  6. 2024Rainbow Teaming: Open-Ended Generation of Diverse Adversarial Promptscited by 189
  7. 2024Do Membership Inference Attacks Work on Large Language Models?cited by 188
  8. 2023Beyond Memorization: Violating Privacy Via Inference with Large Language Modelscited by 167
  9. 2019Show Your Work: Improved Reporting of Experimental Resultscited by 164
  10. 2023Sociotechnical Safety Evaluation of Generative AI Systemscited by 154
  11. 2024Stealing Part of a Production Language Modelcited by 152
  12. 2024LLM Defenses Are Not Robust to Multi-Turn Human Jailbreaks Yetcited by 145
  13. 2023TaskBench: Benchmarking Large Language Models for Task Automationcited by 138
  14. 2022Power to the People? Opportunities and Challenges for Participatory AIcited by 137
  15. 2022Sociotechnical Harms of Algorithmic Systems: Scoping a Taxonomy for Harm Reductioncited by 137
  16. 2020Participation is not a Design Fix for Machine Learningcited by 110
  17. 2020Sponge Examples: Energy-Latency Attacks on Neural Networkscited by 108
  18. 2023Task Contamination: Language Models May Not Be Few-Shot Anymorecited by 98
  19. 2022The Fallacy of AI Functionalitycited by 88
  20. 2025Which Economic Tasks are Performed with AI? Evidence from Millions of Claude Conversationscited by 88
  21. 2022A Deeper Look into Aleatoric and Epistemic Uncertainty Disentanglementcited by 87
  22. 2019Fairwashing: the risk of rationalizationcited by 81
  23. 2023Calibration in Deep Learning: A Survey of the State-of-the-Artcited by 72
  24. 2023FELM: Benchmarking Factuality Evaluation of Large Language Modelscited by 66
  25. 2022Leakage and the Reproducibility Crisis in ML-based Sciencecited by 63
  26. 2021Just What do You Think You're Doing, Dave?' A Checklist for Responsible Data Use in NLPcited by 47
  27. 2023The History and Risks of Reinforcement Learning and Human Feedbackcited by 42
  28. 2020The Problem with Metrics is a Fundamental Problem for AIcited by 37
  29. 2022On the Impossible Safety of Large AI Modelscited by 27
  30. 2021Hard Choices in Artificial Intelligencecited by 26
  31. 2020Overcoming Failures of Imagination in AI Infused System Development and Deploymentcited by 22
  32. 2023Concrete Problems in AI Safety, Revisitedcited by 21
  33. 2022REAL ML: Recognizing, Exploring, and Articulating Limitations of Machine Learning Researchcited by 20
  34. 2024A Collaborative, Human-Centred Taxonomy of AI, Algorithmic, and Automation Harmscited by 19
  35. 2024From Representational Harms to Quality-of-Service Harms: A Case Study on Llama 2 Safety Safeguardscited by 17
  36. 2024Harmful Speech Detection by Language Models Exhibits Gender-Queer Dialect Biascited by 14
  37. 2024A Shared Standard for Valid Measurement of Generative AI Systems' Capabilities, Risks, and Impactscited by 13
  38. 2025Atlas of AI Risks: Enhancing Public Understanding of AI Riskscited by 7

50 most cited of 54 All 54 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 8; 2026: 10.

18 papers.

Sort:

By area:

  1. 2026OneMillion-Bench: How Far are Language Agents from Human Experts?cited by 13
  2. 2025VERA-MH Concept Papercited by 4
  3. 2025Red Teaming AI Red Teamingcited by 3
  4. 2025Ask What Your Country Can Do For You: Towards a Public Red Teaming Modelcited by 1
  5. 2025From Symptoms to Systems: An Expert-Guided Approach to Understanding Risks of Generative AI for Eating Disorderscited by 1
  6. 2026CIRCLE: A Framework for Evaluating AI from a Real-World Lenscited by 1
  7. 2026Making AI Evaluation Deployment Relevant Through Context Specificationcited by 1
  8. 2025Dimensional Characterization and Pathway Modeling for Catastrophic AI Riskscited by 0
  9. 2025Branching Out: Broadening AI Measurement and Evaluation with Measurement Treescited by 0
  10. 2025Towards Real-World Validity in Generative AI Benchmarks: Understanding and Designing Domain-Centered Evaluations for Journalism Practitionerscited by 0
  11. 2025The AI Consumer Index (ACE)cited by 0
  12. 2026Red Teaming LLMs as Socio-Technical Practice: From Exploration and Data Creation to Evaluationcited by 0
6 more
  1. 2026RCTs for Frontier AI Governance: Methodological Challenges and Solutions for Human Uplift Studiescited by 0
  2. 2026Real-World AI Evaluation: How FRAME Generates Systematic Evidence to Resolve the Decision-Maker's Dilemmacited by 0
  3. 2026The Foreign Policy AI Evaluation Gapcited by 0
  4. 2026Toward Human Rights Benchmarking for LLMs: A Pilot Methodologycited by 0
  5. 2026An Investigation of the NeurIPS and ICML 2025 Position Trackscited by 0
  6. 2026ContractScrub: A benchmark for final review of legal contractscited by 0