mlsanityData to 7 Oct 2026

ProactBench: Beyond What The User Asked For

arXiv 2605.09228 ↗First posted 9 May 2026cs.LG cs.AIcited by 4

Not on the map. Why?

Abstract (arXiv)

Most LLM benchmarks score how well a model responds to explicit requests. They leave unmeasured a different conversational ability: noticing and acting on needs the user has implied but not said. We call this conversational proactivity. ProactBench decomposes it into three phase-tied types: textscEmergent, inference from a single disclosed anchor; textscCritical, synthesis across multiple anchors; and textscRecovery, grounded forward-looking value after task completion. We operationalise the benchmark with three agents: a Planner, a User Agent, and an Assistant Model. Their information asymmetries defend against style-confounded scoring, rubric leakage, external-context contamination, and information dumps. The released corpus contains 198 curated dialogues with 624 trigger points across 24 communication styles drawn from a psychometric inventory and audited by an independent LLM judge. Across 16 frontier and open-weight models, textscRecovery is both difficult and weakly predicted by six standard benchmarks, making it a useful new evaluation signal.

Key facts. Cited by 4 papers in the map (data through 7 Oct 2026). Builds on 40 of its 105 references that are in the map. First posted on arXiv 9 May 2026.

Builds on

40 of 105 references.

References · 40

  1. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  2. 2025Qwen3 Technical Reportcited by 8,195
  3. 2020Measuring Massive Multitask Language Understandingcited by 7,768
  4. 2024Qwen2.5 Technical Reportcited by 4,713
  5. 2023Lost in the Middle: How Language Models Use Long Contextscited by 3,383
  6. 2023SWE-bench: Can Language Models Resolve Real-World GitHub Issues?cited by 3,193
  7. 2023GPQA: A Graduate-Level Google-Proof Q&A Benchmarkcited by 2,748
  8. 2022Emergent Abilities of Large Language Modelscited by 2,483
  9. 2024LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Codecited by 1,906
  10. 2023LongBench: A Bilingual, Multitask Benchmark for Long Context Understandingcited by 1,378
  11. 2018Datasheets for Datasetscited by 1,371
  12. 2024RULER: What's the Real Context Size of Your Long-Context Language Models?cited by 1,270
28 more
  1. 2024Chatbot Arena: An Open Platform for Evaluating LLMs by Human Preferencecited by 1,151
  2. 2023Instruction-Following Evaluation for Large Language Modelscited by 1,143
  3. 2023Large Language Models are not Fair Evaluatorscited by 1,050
  4. 2023Towards Understanding Sycophancy in Language Modelscited by 988
  5. 2022Discovering Language Model Behaviors with Model-Written Evaluationscited by 805
  6. 2024LLM Evaluators Recognize and Favor Their Own Generationscited by 651
  7. 2023Are Emergent Abilities of Large Language Models a Mirage?cited by 515
  8. 2023SOTOPIA: Interactive Evaluation for Social Intelligence in Language Agentscited by 350
  9. 2024tinyBenchmarks: evaluating LLMs with fewer examplescited by 295
  10. 2024WildBench: Benchmarking LLMs with Challenging Tasks from Real Users in the Wildcited by 211
  11. 2023Simple synthetic data reduces sycophancy in large language modelscited by 174
  12. 2023Benchmarking Foundation Models with Language-Model-as-an-Examinercited by 172
  13. 2024Quantifying the Persona Effect in LLM Simulationscited by 164
  14. 2023FANToM: A Benchmark for Stress-testing Machine Theory of Mind in Interactionscited by 162
  15. 2024Large Language Models are Inconsistent and Biased Evaluatorscited by 146
  16. 2024STaR-GATE: Teaching Language Models to Ask Clarifying Questionscited by 99
  17. 2023Eliciting Human Preferences with Language Modelscited by 90
  18. 2023Sensitivity, Performance, Robustness: Deconstructing the Effect of Sociodemographic Promptingcited by 80
  19. 2024Croissant: A Metadata Format for ML-Ready Datasetscited by 79
  20. 2024Tell Me More! Towards Implicit User Intention Understanding of Language Model Driven Agentscited by 71
  21. 2025UserBench: An Interactive Gym Environment for User-Centric Agentscited by 69
  22. 2024ClarQ-LLM: A Benchmark for Models Clarifying and Requesting Information in Task-Oriented Dialogcited by 20
  23. 2026ProAgentBench: Evaluating LLM Agents for Proactive Assistance with Real-World Datacited by 15
  24. 2026Proactive Agent Research Environment: Simulating Active Users to Evaluate Proactive Assistantscited by 14
  25. 2024ProCIS: A Benchmark for Proactive Retrieval in Conversationscited by 11
  26. 2025ProactiveEval: A Unified Evaluation Framework for Proactive Dialogue Agentscited by 9
  27. 2025Beyond Reactivity: Measuring Proactive Problem Solving in LLM Agentscited by 9
  28. 2026PROPER Agents: Proactivity Driven Personalized Agents for Advancing Knowledge Gap Navigationcited by 2

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 4.

4 papers.

  1. 2026Proactive Service Agents: A Unified Decision Framework, Methods, and Evaluationcited by 2
  2. 2026NeMo Data Designer: An Extensible Framework for Multimodal Synthetic Data Generationcited by 0
  3. 2026Asking for What Was Never Requested: Horizontal and Vertical Proactivity in Agentscited by 0
  4. 2026Foundations of Proactive Agents: Principles, Technical Layers, and Proactivity-Gymcited by 0