mlsanityData to 7 Oct 2026

AI Evaluation Should Work With Humans

arXiv 2608.13577 ↗First posted 6 Jul 2026cs.AI cs.LGcited by 0

Not on the map. Why?

Abstract (arXiv)

This position paper argues that the dominant paradigm of AI evaluation (which focuses on superhuman autonomous performance and so implicitly targets the goal of replacing humans) is guiding AI development in the wrong direction. Instead, the AI community should pivot to evaluating the performance of human--AI teams. We argue that this collaborative shift will foster AI systems that act as true complements to human capabilities and therefore lead to far better societal outcomes than will the current process.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 22 of its 62 references that are in the map. First posted on arXiv 6 Jul 2026.

Builds on

22 of 62 references.

References · 22

  1. 2014ImageNet Large Scale Visual Recognition Challengecited by 8,441
  2. 2020Measuring Massive Multitask Language Understandingcited by 7,768
  3. 2016"Why Should I Trust You?": Explaining the Predictions of Any Classifiercited by 5,040
  4. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,854
  5. 2019SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systemscited by 1,718
  6. 2024Chatbot Arena: An Open Platform for Evaluating LLMs by Human Preferencecited by 1,151
  7. 2019On the Utility of Learning about Humans for Human-AI Coordinationcited by 345
  8. 2018Categorizing Variants of Goodhart's Lawcited by 132
  9. 2020Uncertainty as a Form of Transparency: Measuring, Communicating, and Using Uncertaintycited by 128
  10. 2025LLM Social Simulations Are a Promising Research Methodcited by 97
  11. 2023Clarify When Necessary: Resolving Ambiguity Through Interaction with LMscited by 93
  12. 2024Human Creativity in the Age of LLMs: Randomized Experiments on Divergent and Convergent Thinkingcited by 43
10 more
  1. 2025Gradual Disempowerment: Systemic Existential Risks from Incremental AI Developmentcited by 42
  2. 2021Evaluation of Human-AI Teams for Learned and Rule-Based Agents in Hanabicited by 37
  3. 2026This human study did not involve human subjects: Validating LLM simulations as behavioral evidencecited by 26
  4. 2021Evaluating the Robustness of Collaborative Agentscited by 25
  5. 2024Evaluating Human-AI Collaboration: A Review and Methodological Frameworkcited by 25
  6. 2023Investigating Agency of LLMs in Human-AI Collaboration Taskscited by 15
  7. 2024Explainable AI improves task performance in human-AI collaborationcited by 11
  8. 2024AI-enhanced Collective Intelligencecited by 10
  9. 2022Deep Quality Estimation: Creating Surrogate Models for Human Quality Ratingscited by 6
  10. 2025AI for Just Work: Constructing Diverse Imaginations of AI beyond "Replacing Humans"cited by 2

Built on by

Not cited yet.