mlsanityData to 14 Oct 2026

Rethinking Model Evaluation as Narrowing the Socio-Technical Gap

arXiv 2306.03100 ↗First posted 1 Jun 2023cs.HC cs.AIcited by 56

On the map: LLM reasoning → Reasoning benchmarks and evaluation

Abstract (arXiv)

The recent development of generative large language models (LLMs) poses new challenges for model evaluation that the research community and industry have been grappling with. While the versatile capabilities of these models ignite much excitement, they also inevitably make a leap toward homogenization: powering a wide range of applications with a single, often referred to as ``general-purpose'', model. In this position paper, we argue that model evaluation practices must take on a critical task to cope with the challenges and responsibilities brought by this homogenization: providing valid assessments for whether and how much human needs in diverse downstream use cases can be satisfied by the given model (textitsocio-technical gap). By drawing on lessons about improving research realism from the social sciences, human-computer interaction (HCI), and the interdisciplinary field of explainable AI (XAI), we urge the community to develop evaluation methods based on real-world contexts and human requirements, and embrace diverse evaluation methods with an acknowledgment of trade-offs between realisms and pragmatic costs to conduct the evaluation. By mapping HCI and current NLG evaluation methods, we identify opportunities for evaluation methods for LLMs to narrow the socio-technical gap and pose open questions.

Key facts. Cited by 56 papers in the map (data through 14 Oct 2026). Placed in Reasoning benchmarks and evaluation, one of 17 directions in LLM reasoning. Builds on 24 of its 57 references that are in the map. First posted on arXiv 1 Jun 2023.

Builds on

24 of 57 references.

References · 24

  1. 2022Training language models to follow instructions with human feedbackcited by 15,971
  2. 2019BERTScore: Evaluating Text Generation with BERTcited by 5,512
  3. 2022Emergent Abilities of Large Language Modelscited by 2,512
  4. 2022Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language modelscited by 1,998
  5. 2017Towards A Rigorous Science of Interpretable Machine Learningcited by 1,416
  6. 2015Teaching Machines to Read and Comprehendcited by 1,049
  7. 2016Abstractive Text Summarization Using Sequence-to-Sequence RNNs and Beyondcited by 812
  8. 2020SummEval: Re-evaluating Summarization Evaluationcited by 674
  9. 2019SAMSum Corpus: A Human-annotated Dialogue Dataset for Abstractive Summarizationcited by 535
  10. 2021AI and the Everything in the Whole Wide World Benchmarkcited by 349
  11. 2021All That's 'Human' Is Not Gold: Evaluating Human Evaluation of Generated Textcited by 287
  12. 2020Questioning the AI: Informing Design Practices for Explainable AI User Experiencescited by 253
12 more
  1. 2020A Survey of Evaluation Metrics Used for NLG Systemscited by 164
  2. 2022Repairing the Cracked Foundation: A Survey of Obstacles in Evaluation Practices for Generated Textcited by 155
  3. 2021Human-Centered Explainable AI (XAI): From Algorithms to User Experiencescited by 123
  4. 2021Towards a Science of Human-AI Decision Making: A Survey of Empirical Studiescited by 86
  5. 2020Human Factors in Model Interpretability: Industry Practices, Challenges, and Needscited by 74
  6. 2021Beyond Expertise and Roles: A Framework to Characterize the Stakeholders of Interpretable Machine Learning and their Needscited by 64
  7. 2020Evaluating Conversational Recommender Systems via User Simulationcited by 47
  8. 2022Connecting Algorithmic Research and Usage Contexts: A Perspective of Contextualized Evaluation for Explainable AIcited by 40
  9. 2019Ask Not What AI Can Do, But What AI Should Do: Towards a Framework of Task Delegabilitycited by 36
  10. 2022Deconstructing NLG Evaluation: Evaluation Practices, Assumptions, and Their Implicationscited by 32
  11. 2022Use-Case-Grounded Simulations for Explanation Evaluationcited by 19
  12. 2023Designerly Understanding: Information Needs for Model Transparency to Support Design Ideation for AI-Powered User Experiencecited by 18

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 3; 2024: 9; 2025: 28; 2026: 16.

56 papers.

Sort:

By area:

  1. 2023Does Writing with Language Models Reduce Content Diversity?cited by 192
  2. 2023Sociotechnical Safety Evaluation of Generative AI Systemscited by 157
  3. 2024BetterBench: Assessing AI Benchmarks, Uncovering Issues, and Establishing Best Practicescited by 114
  4. 2023Identifying and Mitigating the Security Risks of Generative AIcited by 91
  5. 2025Can We Trust AI Benchmarks? An Interdisciplinary Review of Current Issues in AI Evaluationcited by 60
  6. 2024Benchmarks as Microscopes: A Call for Model Metrologycited by 35
  7. 2024Towards interactive evaluations for interaction harms in human-AI systemscited by 29
  8. 2025Amulet: ReAlignment During Test Time for Personalized Preference Adaptation of LLMscited by 29
  9. 2024ECBD: Evidence-Centered Benchmark Design for NLPcited by 28
  10. 2025Understanding the LLM-ification of CHI: Unpacking the Impact of LLMs at CHI through a Systematic Literature Reviewcited by 16
  11. 2025Rethinking Evaluation in the Era of Time Series Foundation Models: (Un)known Information Leakage Challengescited by 16
  12. 2024What the Harm? Quantifying the Tangible Impact of Gender Bias in Machine Translation with a Human-centered Studycited by 15
38 more
  1. 2025The AI Gap: How Socioeconomic Status Affects Language Technology Interactionscited by 15
  2. 2025VeriLA: A Human-Centered Evaluation Framework for Interpretable Verification of LLM Agent Failurescited by 14
  3. 2024Generative Echo Chamber? Effects of LLM-Powered Search Systems on Diverse Information Seekingcited by 13
  4. 2024The Responsible Foundation Model Development Cheatsheet: A Review of Tools & Resourcescited by 11
  5. 2025Evaluating Multimodal Language Models as Visual Assistants for Visually Impaired Userscited by 11
  6. 2025SPHERE: An Evaluation Card for Human-AI Systemscited by 9
  7. 2025Translation in the Hands of Many:Centering Lay Users in Machine Translation Interactionscited by 8
  8. 2025Doubly-Robust LLM-as-a-Judge: Externally Valid Estimation with Imperfect Personascited by 8
  9. 2025Benchmarking is Broken -- Don't Let AI be its Own Judgecited by 8
  10. 2026Computational Hermeneutics: Evaluating generative AI as a cultural technologycited by 8
  11. 2024Faux Polyglot: A Study on Information Disparity in Multilingual Large Language Modelscited by 7
  12. 2025Browsing Lost Unformed Recollections: A Benchmark for Tip-of-the-Tongue Search and Reasoningcited by 7
  13. 2025Measuring What Matters: Connecting AI Ethics Evaluations to System Attributes, Hazards, and Harmscited by 5
  14. 2026Emergence WebVoyager: Toward Consistent and Transparent Evaluation of (Web) Agents in The Wildcited by 5
  15. 2024(Beyond) Reasonable Doubt: Challenges that Public Defenders Face in Scrutinizing AI in Courtcited by 3
  16. 2025A Survey on Human-Centered Evaluation of Explainable AI Methods in Clinical Decision Support Systemscited by 3
  17. 2025DICE: A Framework for Dimensional and Contextual Evaluation of Language Modelscited by 3
  18. 2025Distinguishing Task-Specific and General-Purpose AI in Regulationcited by 3
  19. 2025ProxAnn: Use-Oriented Evaluations of Topic Models and Document Clusteringcited by 3
  20. 2025Modeling Motivated Reasoning in Law: Evaluating Strategic Role Conditioning in LLM Summarizationcited by 3
  21. 2025Risk Management for Mitigating Benchmark Failure Modes: BenchRiskcited by 3
  22. 2025TS-Arena -- A Live Forecast Pre-Registration Platformcited by 3
  23. 2026AI Evaluation Should Require Standardized Item-Level Data Releasescited by 3
  24. 2026Position: Evaluation Scores Are Perishable Knowledge Claimscited by 3
  25. 2025Does Acceleration Cause Hidden Instability in Vision Language Models? Uncovering Instance-Level Divergence Through a Large-Scale Empirical Studycited by 2
  26. 2025ParaStudent: Closing the Sim2Real Gap in User Simulators for AI Tutor Evaluationcited by 2
  27. 2025From Queries to Criteria: Understanding How Astronomers Evaluate LLMscited by 2
  28. 2025Who's Asking? Simulating Role-Based Questions for Conversational AI Evaluationcited by 2
  29. 2026CIRCLE: A Framework for Evaluating AI from a Real-World Lenscited by 2
  30. 2025Unfettered Forceful Skill Acquisition with Physical Reasoning and Coordinate Frame Labelingcited by 1
  31. 2026Reheat Nachos for Dinner? Evaluating AI Support for Cross-Cultural Communication of Neologismscited by 1
  32. 2026Does Theory of Mind Improvement Really Benefit Human-AI Interactions? Empirical Findings from Interactive Evaluationscited by 1
  33. 2025On the Dual-Use Dilemma in Physical Reasoning and Forcecited by 0
  34. 2025A Conceptual Framework for AI Capability Evaluationscited by 0
  35. 2025Towards Real-World Validity in Generative AI Benchmarks: Understanding and Designing Domain-Centered Evaluations for Journalism Practitionerscited by 0
  36. 2026Real-World AI Evaluation: How FRAME Generates Systematic Evidence to Resolve the Decision-Maker's Dilemmacited by 0
  37. 2026Results-Actionability Gap: Understanding How Practitioners Evaluate LLM Products in the Wildcited by 0
  38. 2026Beyond Third-Person Audits: Situated Interaction Auditing for User-Centered LLM Bias Researchcited by 0

50 most cited of 56 All 56 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024ConSiDERS-The-Human Evaluation Framework: Rethinking Human Evaluation for Generative Large Language Models
  2. 2024Can Large Language Models be Trusted for Evaluation? Scalable Meta-Evaluation of LLMs as Evaluators via Agent Debate
  3. 2024DnA-Eval: Enhancing Large Language Model Evaluation through Decomposition and Aggregation
  4. 2023Who's Thinking? A Push for Human-Centered Evaluation of LLMs using the XAI Playbook
  5. 2023Exploring the Reliability of Large Language Models as Customized Evaluators for Diverse NLP Tasks
  6. 2023Establishing Trustworthiness: Rethinking Tasks and Model Evaluation
  7. 2024Under the Surface: Tracking the Artifactuality of LLM-Generated Data
  8. 2025Neither Valid nor Reliable? Investigating the Use of LLMs as Judges