mlsanityData to 14 Oct 2026

Beyond Reactivity: Measuring Proactive Problem Solving in LLM Agents

arXiv 2510.19771 ↗First posted 22 Oct 2025cs.AIcited by 10

On the map: LLM agents and tool use → General LLM agent frameworks and benchmarks

Abstract (arXiv)

LLM-based agents are increasingly moving towards proactivity: rather than awaiting instruction, they exercise agency to anticipate user needs and solve them autonomously. However, evaluating proactivity is challenging; current benchmarks are constrained to localized context, limiting their ability to test reasoning across sources and longer time horizons. To address this gap, we present PROBE (Proactive Resolution Of BottlEnecks). PROBE decomposes proactivity as a pipeline of three core capabilities: (1) searching for unspecified issues, (2) identifying specific bottlenecks, and (3) executing appropriate resolutions. We apply PROBE to evaluate leading LLMs and popular agentic frameworks, showing that even state-of-the-art models struggle to solve this benchmark. Computing our consistent measurements across frontier LLMs and agents, we find that the best end-to-end performance of 40% is achieved by both GPT-5 and Claude Opus-4.1. Additionally, we demonstrate the relative capabilities of each model and analyze mutual failure modes. Our results highlight the current limitations of autonomous action in agentic systems, and expose promising future research directions.

Key facts. Cited by 10 papers in the map (data through 14 Oct 2026). Placed in General LLM agent frameworks and benchmarks, one of 16 directions in LLM agents and tool use. Builds on 23 of its 30 references that are in the map. First posted on arXiv 22 Oct 2025.

Builds on

23 of 30 references.

Closest ancestors · 4

Same direction, newest first

  1. 2024Proactive Agent: Shifting LLM Agents from Reactive Responses to Active Assistance16 Oct
  2. 2024WebPilot: A Versatile and Autonomous Multi-Agent System for Web Task Execution with Strategic Exploration28 Aug
  3. 2024Ask-before-Plan: Proactive Language Agents for Real-World Planning18 Jun
  4. 2023ProAgent: Building Proactive Cooperative Agents with Large Language Models22 Aug

Other references · 19

  1. 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,695
  2. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
  3. 2023Reflexion: Language Agents with Verbal Reinforcement Learningcited by 4,276
  4. 2023Toolformer: Language Models Can Teach Themselves to Use Toolscited by 4,182
  5. 2023SWE-bench: Can Language Models Resolve Real-World GitHub Issues?cited by 3,373
  6. 2023ToolLLM: Facilitating Large Language Models to Master 16000+ Real-world APIscited by 1,947
  7. 2024SWE-agent: Agent-Computer Interfaces Enable Automated Software Engineeringcited by 1,737
  8. 2024GPT-4V(ision) is a Generalist Web Agent, if Groundedcited by 563
  9. 2024Agent S: An Open Agentic Framework that Uses Computers Like a Humancited by 187
  10. 2024MDAgents: An Adaptive Collaboration of LLMs for Medical Decision-Makingcited by 163
  11. 2025Synthetic Data Generation Using Large Language Models: Advances in Text and Codecited by 61
  12. 2025ContextAgent: Context-Aware Proactive LLM Agents with Open-World Sensory Perceptionscited by 45
7 more
  1. 2025Creating General User Models from Computer Usecited by 44
  2. 2023VisionTasker: Mobile Task Automation Using Vision Based UI Understanding and LLM Task Planningcited by 28
  3. 2025ProactiveVideoQA: A Comprehensive Benchmark Evaluating Proactive Interactions in Video Large Language Modelscited by 27
  4. 2023The Looming Threat of Fake and LLM-generated LinkedIn Profiles: Challenges and Opportunities for Detection and Preventioncited by 18
  5. 2025ProactiveVA: Proactive Visual Analytics with LLM-Based UI Agentcited by 14
  6. 2024ProCIS: A Benchmark for Proactive Retrieval in Conversationscited by 11
  7. 2025SPARTA ALIGNMENT: Collectively Aligning Multiple Language Models through Combatcited by 9

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 10.

10 papers.

  1. 2026Dive into Claude Code: The Design Space of Today's and Future AI Agent Systemscited by 38
  2. 2026ProactBench: Beyond What The User Asked Forcited by 4
  3. 2026Agentic Coding Needs Proactivity, Not Just Autonomycited by 3
  4. 2026Knowing Isn't Understanding: Re-grounding Generative Proactivity with Epistemic and Behavioral Insightcited by 2
  5. 2026Proactive Service Agents: A Unified Decision Framework, Methods, and Evaluationcited by 2
  6. 2026TIDE: Proactive Multi-Problem Discovery via Template-Guided Iterationcited by 1
  7. 2026Agents in the Large: Perception-Centered Architecture for Persistent Agentscited by 1
  8. 2026Just A Rather Very Intelligent Spoken Agentcited by 0
  9. 2026Foundations of Proactive Agents: Principles, Technical Layers, and Proactivity-Gymcited by 0
  10. 2026Jarvis: A Proactive Speech Agent for Multi-Party Conversationscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025GroundAct: Can LLM Agents Ground Actions in Environmental States?
  2. 2026AgentAbstain: Do LLM Agents Know When Not to Act?
  3. 2026AgentEscapeBench: Evaluating Out-of-Domain Tool-Grounded Reasoning in LLM Agents
  4. 2026From Question Answering to Task Completion: A Survey on Agent System and Harness Design
  5. 2024AgentQuest: A Modular Benchmark Framework to Measure Progress and Improve LLM Agents
  6. 2026Agentic CLEAR: Automating Multi-Level Evaluation of LLM Agents
  7. 2026From Task Solving to Robust Real-World Adaptation in LLM Agents
  8. 2025LLM Agents Beyond Utility: An Open-Ended Perspective