mlsanityData to 14 Oct 2026

ChatBench: From Static Benchmarks to Human-AI Evaluation

arXiv 2504.07114 ↗First posted 22 Mar 2025cs.CL cs.AI cs.CYcited by 34

Not on the map. Why?

Abstract (arXiv)

With the rapid adoption of LLM-based chatbots, there is a pressing need to evaluate what humans and LLMs can achieve together. However, standard benchmarks, such as MMLU, measure LLM capabilities in isolation (i.e., "AI-alone"). Here, we design and conduct a user study to convert MMLU questions into user-AI conversations, by seeding the user with the question and having them carry out a conversation with the LLM to answer their question. We release ChatBench, a new dataset with AI-alone, user-alone, and user-AI data for 396 questions and two LLMs, including 144K answers and 7,336 user-AI conversations. We find that AI-alone accuracy fails to predict user-AI accuracy, with significant differences across multiple subjects (math, physics, and moral reasoning), and we analyze the user-AI conversations to provide insight into how they diverge from AI-alone benchmarks. Finally, we show that fine-tuning a user simulator on a subset of ChatBench improves its ability to estimate user-AI accuracies, increasing correlation on held-out questions by more than 20 points, creating possibilities for scaling interactive evaluation.

Key facts. Cited by 34 papers in the map (data through 14 Oct 2026). Builds on 28 of its 52 references that are in the map. First posted on arXiv 22 Mar 2025.

Builds on

28 of 52 references.

References · 28

  1. 2024The Llama 3 Herd of Modelscited by 14,715
  2. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  3. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
  4. 2020Measuring Massive Multitask Language Understandingcited by 7,963
  5. 2018HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answeringcited by 3,754
  6. 2024Chatbot Arena: An Open Platform for Evaluating LLMs by Human Preferencecited by 1,184
  7. 2022Out of One, Many: Using Language Models to Simulate Human Samplescited by 752
  8. 2023AlpacaFarm: A Simulation Framework for Methods that Learn from Human Feedbackcited by 748
  9. 2024WildChat: 1M ChatGPT Interaction Logs in the Wildcited by 629
  10. 2023LMSYS-Chat-1M: A Large-Scale Real-World LLM Conversation Datasetcited by 461
  11. 2020Does the Whole Exceed its Parts? The Effect of AI Explanations on Complementary Team Performancecited by 399
  12. 2024WildBench: Benchmarking LLMs with Challenging Tasks from Real Users in the Wildcited by 218
16 more
  1. 2023Marked Personas: Using Natural Language Prompts to Measure Stereotypes in Language Modelscited by 207
  2. 2024MediQ: Question-Asking LLMs and a Benchmark for Reliable Interactive Clinical Reasoningcited by 154
  3. 2024Adding Error Bars to Evals: A Statistical Approach to Language Model Evaluationscited by 148
  4. 2024Large language models that replace human participants can harmfully misportray and flatten identity groupscited by 144
  5. 2023Aligning Language Models to User Opinionscited by 99
  6. 2023CoMPosT: Characterizing and Evaluating Caricature in LLM Simulationscited by 97
  7. 2023Evaluating Language Models for Mathematics through Interactionscited by 70
  8. 2025Language Model Fine-Tuning on Scaled Survey Data for Predicting Distributions of Public Opinionscited by 66
  9. 2024Collaborative Gym: A Framework for Enabling and Evaluating Human-Agent Collaborationcited by 65
  10. 2024Towards Responsible Development of Generative AI for Education: An Evaluation-Driven Approachcited by 54
  11. 2024Shaping Human-AI Collaboration: Varied Scaffolding Levels in Co-writing with Language Modelscited by 51
  12. 2023PlatoLM: Teaching LLMs in Multi-Round Dialogue via a User Simulatorcited by 29
  13. 2024BASES: Large-scale Web Search User Simulation with Large Language Model based Agentscited by 25
  14. 2024IQA-EVAL: Automatic Evaluation of Human-Model Interactive Question Answeringcited by 19
  15. 2024More than Marketing? On the Information Value of AI Benchmarks for Practitionerscited by 17
  16. 2018Characterizing the Global Crowd Workforce: A Cross-Country Comparison of Crowdworker Demographicscited by 10

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 9; 2026: 25.

34 papers.

Sort:

By area:

  1. 2025LLMs Get Lost In Multi-Turn Conversationcited by 429
  2. 2026Lost in Simulation: LLM-Simulated Users are Unreliable Proxies for Human Users in Agentic Evaluationscited by 40
  3. 2026HumanLM: Simulating Users with State Alignment Beats Response Imitationcited by 37
  4. 2025User Feedback in Human-LLM Dialogues: A Lens to Understand Users But Noisy as a Learning Signalcited by 19
  5. 2026HorizonBench: Long-Horizon Personalization with Evolving Preferencescited by 17
  6. 2025Potemkin Understanding in Large Language Modelscited by 15
  7. 2025Generative Value Conflicts Reveal LLM Prioritiescited by 13
  8. 2026Quantifying the Utility of User Simulators for Building Collaborative LLM Assistantscited by 10
  9. 2025LLMs Outperform Experts on Challenging Biology Benchmarkscited by 9
  10. 2025The Inadequacy of Offline LLM Evaluations: A Need to Account for Personalization in Model Behaviorcited by 8
  11. 2026MatrAIx: Simulating the World with 8.3 Billion Persona Agentscited by 7
  12. 2025Mitigating Lost in Multi-turn Conversation via Curriculum RL with Verifiable Accuracy and Abstention Rewardscited by 6
22 more
  1. 2026DiscoverLLM: From Executing Intents to Discovering Themcited by 6
  2. 2025Exploiting Leaderboards for Large-Scale Distribution of Malicious Modelscited by 5
  3. 2025ELI-Why: Evaluating the Pedagogical Utility of Language Model Explanationscited by 4
  4. 2026Understanding Persuasive Interactions between Generative Social Agents and Humans: The Knowledge-based Persuasion Model (KPM)cited by 3
  5. 2026GAUGE: When Not to Trust LLM-as-a-Judge in User-Simulated Evaluation of Task-Oriented Agentscited by 3
  6. 2026Cultural Compass: A Framework for Organizing Societal Norms to Detect Violations in Human-AI Conversationscited by 2
  7. 2026Synthetic Users, Real Differences: an Evaluation Framework for User Simulation in Multi-Turn Conversationscited by 2
  8. 2026ReIn: Conversational Error Recovery with Reasoning Inceptioncited by 1
  9. 2026Does Theory of Mind Improvement Really Benefit Human-AI Interactions? Empirical Findings from Interactive Evaluationscited by 1
  10. 2026Enjoy Your Talk: A Human-Centered Benchmark for Multi-Turn Dialogue with Decoupled User Simulation, Target Modeling, and Judgingcited by 1
  11. 2026KnowSim: Evaluating Information Calibration in LLM Assistants with User Simulators that Learncited by 1
  12. 2026CentaurBench: Benchmarking LLM Capabilities on Augmenting vs. Automating Real-World Work Taskscited by 1
  13. 2026From Task Success to Productive Success: Evaluating Human-AI Collaboration by Quality and Costcited by 1
  14. 2026A Vision for Multisensory Intelligence: Sensing, Science, and Synergycited by 0
  15. 2026From Feelings to Metrics: Understanding and Formalizing How Users Vibe-Test LLMscited by 0
  16. 2026API Benchmark Scores Do Not Reliably Transfer to Chatbot Interfacescited by 0
  17. 2026ORQA: An Occupation-Realistic Question and Answer Framework for LLM Professional Knowledgecited by 0
  18. 2026LLMs as Oracles: Reliance on LLMs for Subjective Personal Questionscited by 0
  19. 2026Available but Unclaimed: An Empirical Study of Human-AI Synergycited by 0
  20. 2026CUEing User Simulators: Calibrated User Embeddings for Multi-Turn Benchmarkingcited by 0
  21. 2026Dataset Signatures in Human-LLM Interactions and User Modelingcited by 0
  22. 2026Training User Simulators with Latent Intrinsic Feedbackcited by 0