mlsanityData to 14 Oct 2026

Ask Early, Ask Late, Ask Right: When Does Clarification Timing Matter for Long-Horizon Agents?

arXiv 2605.07937 ↗First posted 8 May 2026cs.CLcited by 4

Not on the map. Why?

Abstract (arXiv)

Long-horizon AI agents execute complex workflows spanning hundreds of sequential actions, yet a single wrong assumption early on can cascade into irreversible errors. When instructions are incomplete, the agent must decide not only whether to ask for clarification but when, and no prior work measures how clarification value changes over the course of execution. We introduce a forced-injection framework that provides ground-truth clarifications at controlled points in the agent's trajectory across four information dimensions (goal, input, constraint, context), three agent benchmarks, and four frontier models (three per benchmark; one on a single benchmark only; 84 task variants; 6,000+ runs). Counter to the common intuition that "earlier is always better," we find that the value of clarification depends sharply on what information is missing: goal clarification loses nearly all value after 10% of execution (pass@3 drops from 0.78 to baseline), while input clarification retains value through roughly 50%. Deferring any clarification type past mid-trajectory degrades performance below never asking at all. Cross-model Kendall tau correlations (0.78-0.87 among models sharing identical task coverage; 0.34-0.67 across the full 4-model panel) confirm these timing profiles are substantially task-intrinsic. A complementary study of 300 unscripted sessions reveals that no current frontier model asks within the empirically optimal window, with strategies ranging from over-asking (52% of sessions) to never asking at all. These empirical demand curves provide the quantitative foundation that existing theoretical frameworks require but have lacked, and establish concrete design targets for timing-aware clarification policies. Code and data will be publicly released.

Key facts. Cited by 4 papers in the map (data through 14 Oct 2026). Builds on 34 of its 48 references that are in the map. First posted on arXiv 8 May 2026.

Builds on

34 of 48 references.

References · 34

  1. 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,695
  2. 2021Evaluating Large Language Models Trained on Codecited by 8,320
  3. 2023SWE-bench: Can Language Models Resolve Real-World GitHub Issues?cited by 3,373
  4. 2023WebArena: A Realistic Web Environment for Building Autonomous Agentscited by 2,006
  5. 2022Language Models (Mostly) Know What They Knowcited by 1,856
  6. 2024τ-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domainscited by 1,402
  7. 2023AgentBench: Evaluating LLMs as Agentscited by 1,267
  8. 2024OSWorld: Benchmarking Multimodal Agents for Open-Ended Tasks in Real Computer Environmentscited by 1,195
  9. 2023GAIA: a benchmark for General AI Assistantscited by 1,128
  10. 2024AppWorld: A Controllable World of Apps and People for Benchmarking Interactive Coding Agentscited by 401
  11. 2020AmbigQA: Answering Ambiguous Open-domain Questionscited by 390
  12. 2024TheAgentCompany: Benchmarking LLM Agents on Consequential Real World Taskscited by 326
22 more
  1. 2025CollabLLM: From Passive Responders to Active Collaboratorscited by 124
  2. 2018Learning to Ask Good Questions: Ranking Clarification Questions using Neural Expected Value of Perfect Informationcited by 115
  3. 2025UserBench: An Interactive Gym Environment for User-Centric Agentscited by 76
  4. 2022CLAM: Selective Clarification for Ambiguous Questions with Generative Language Modelscited by 69
  5. 2024Ask-before-Plan: Proactive Language Agents for Real-World Planningcited by 60
  6. 2020ConvAI3: Generating Clarifying Questions for Open-Domain Dialogue Systems (ClariQ)cited by 58
  7. 2026MCP-Atlas: A Large-Scale Benchmark for Tool-Use Competency with Real MCP Serverscited by 53
  8. 2025Structured Uncertainty guided Clarification for LLM Agentscited by 41
  9. 2024Uncertainty of Thoughts: Uncertainty-Aware Planning Enhances Information Seeking in Large Language Modelscited by 39
  10. 2025ScaleMCP: Dynamic and Auto-Synchronizing Model Context Protocol Tools for LLM Agentscited by 36
  11. 2022Task Ambiguity in Humans and Language Modelscited by 28
  12. 2025Active Task Disambiguation with LLMscited by 23
  13. 2025MemTool: Optimizing Short-Term Memory Management for Dynamic Tool Calling in LLM Agent Multi-Turn Conversationscited by 21
  14. 2026Don't Break the Cache: An Evaluation of Prompt Caching for Long-Horizon Agentic Taskscited by 19
  15. 2025Tool-to-Agent Retrieval: Bridging Tools and Agents for Scalable LLM Multi-Agent Systemscited by 15
  16. 2026Value of Information: A Framework for Human-Agent Communicationcited by 12
  17. 2026Chronos: Temporal-Aware Conversational Agents with Structured Event Retrieval for Long-Term Memorycited by 8
  18. 2025CaRT: Teaching LLM Agents to Know When They Know Enoughcited by 7
  19. 2026LHAW: Controllable Underspecification for Long-Horizon Taskscited by 6
  20. 2024Avoiding Catastrophe in Online Learning by Asking for Helpcited by 4
  21. 2026PRISM: Festina Lente Proactivity -- Risk-Sensitive, Uncertainty-Aware Deliberation for Proactive Agentscited by 3
  22. 2025Self-Regulation and Requesting Interventionscited by 2

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 4.

4 papers.