mlsanityData to 7 Oct 2026

Ask or Assume? Uncertainty-Aware Clarification-Seeking in Coding Agents

arXiv 2603.26233 ↗First posted 27 Mar 2026cs.CLcited by 15

On the map: LLM agents and tool use → General LLM agent frameworks and benchmarks

Abstract (arXiv)

As Large Language Model (LLM) agents are increasingly deployed in open-ended domains like software engineering, they frequently encounter underspecified instructions that lack crucial context. While human developers naturally resolve underspecification by asking clarifying questions, current agents are largely optimized for autonomous execution. In this work, we systematically evaluate the clarification-seeking abilities of LLM agents on an underspecified variant of SWE-bench Verified. We propose an uncertainty-aware multi-agent scaffold that decouples underspecification detection from code execution. Across both proprietary and open-weight frontier LLMs, our scaffold achieves a 69.40% task resolve rate, significantly outperforming a standard single-agent setup and closing the performance gap with agents operating on fully specified instructions. Furthermore, we find that the multi-agent system exhibits well-calibrated information-seeking behavior, conserving queries on simple tasks while proactively seeking information on more complex issues. These findings indicate that current models can be turned into proactive collaborators, where agents independently recognize when to ask questions to elicit missing information in real-world, underspecified tasks.

Key facts. Cited by 15 papers in the map (data through 7 Oct 2026). Placed in General LLM agent frameworks and benchmarks, one of 16 directions in LLM agents and tool use. Builds on 30 of its 52 references that are in the map. First posted on arXiv 27 Mar 2026.

Builds on

30 of 52 references.

Closest ancestors · 3

Same direction, newest first

  1. 2025Training Proactive and Personalized LLM Agents4 Nov
  2. 2024τ-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domains17 Jun
  3. 2024Aligning Language Models to Explicitly Handle Ambiguity18 Apr

Other references · 27

  1. 2023Generative Agents: Interactive Simulacra of Human Behaviorcited by 3,500
  2. 2023SWE-bench: Can Language Models Resolve Real-World GitHub Issues?cited by 3,193
  3. 2022Language Models (Mostly) Know What They Knowcited by 1,783
  4. 2024OpenHands: An Open Platform for AI Software Developers as Generalist Agentscited by 956
  5. 2023ChatDev: Communicative Agents for Software Developmentcited by 893
  6. 2022Teaching Models to Express Their Uncertainty in Wordscited by 731
  7. 2019Asking Clarifying Questions in Open-Domain Information-Seeking Conversationscited by 136
  8. 2023Decomposing Uncertainty for Large Language Models through Input Clarification Ensemblingcited by 113
  9. 2018Learning to Ask Good Questions: Ranking Clarification Questions using Neural Expected Value of Perfect Informationcited by 110
  10. 2025SWEET-RL: Training Multi-Turn LLM Agents on Collaborative Reasoning Taskscited by 108
  11. 2024Relying on the Unreliable: The Impact of Language Models' Reluctance to Express Uncertaintycited by 106
  12. 2024STaR-GATE: Teaching Language Models to Ask Clarifying Questionscited by 99
15 more
  1. 2023Clarify When Necessary: Resolving Ambiguity Through Interaction with LMscited by 93
  2. 2023Eliciting Human Preferences with Language Modelscited by 90
  3. 2024Modeling Future Conversation Turns to Teach LLMs to Ask Clarifying Questionscited by 70
  4. 2023Goal Driven Discovery of Distributional Differences via Language Descriptionscited by 61
  5. 2024Agents' Room: Narrative Generation through Multi-step Collaborationcited by 57
  6. 2025Flipping the Dialogue: Training and Evaluating User Language Modelscited by 50
  7. 2024Asking the Right Question at the Right Time: Human and Model Uncertainty Guidance to Ask Clarification Questionscited by 32
  8. 2024Bayesian Preference Elicitation with Language Modelscited by 32
  9. 2025Ambig-SWE: Interactive Agents to Overcome Underspecificity in Software Engineeringcited by 28
  10. 2024HumanEvalComm: Benchmarking the Communication Competence of Code Generation for LLMs and LLM Agentcited by 24
  11. 2025Semantic Volume: Quantifying and Detecting both External and Internal Uncertainty in LLMscited by 19
  12. 2022Python Code Generation by Asking Clarification Questionscited by 16
  13. 2024Loose LIPS Sink Ships: Asking Questions in Battleship with Language-Informed Program Samplingcited by 12
  14. 2026ClarEval: A Benchmark for Evaluating Clarification Skills of Code Agents under Ambiguous Instructionscited by 5
  15. 2025Curiosity by Design: An LLM-based Coding Assistant Asking Clarification Questionscited by 3

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 15.

15 papers.

Sort:

By area:

  1. 2026AgentAtlas: Beyond Outcome Leaderboards for LLM Agentscited by 9
  2. 2026ICAE-Bench: Evaluating Coding Agents as Interactive Project Builderscited by 5
  3. 2026SWE-Touch: Benchmarking Coding Agents When Users Touch the Codecited by 2
  4. 2026Proactive Service Agents: A Unified Decision Framework, Methods, and Evaluationcited by 2
  5. 2026ASPI: Seeking Ambiguity Clarification Amplifies Prompt Injection Vulnerability in LLM Agentscited by 1
  6. 2026Don't Start What You Can't Finish: A Counterfactual Audit of Support-State Triage in LLM Agentscited by 0
  7. 2026Agent System Operations: Categorization, Challenges, and Future Directionscited by 0
  8. 2026Turning Intent into Specifications: A Benchmark and an Interactive User-Assistant Agentcited by 0
  9. 2026AcCoRD: Evaluating User-Agent Collaboration Under Realistic User Preference Dynamicscited by 0
  10. 2026RealSWE: A Compositional Evaluation of Coding Agents under Realistic User Requestscited by 0
  11. 2026Uncertainty Quantification for LLM Agents: A Taxonomy, an Evaluation Protocol, and an Empirical Studycited by 0
  12. 2026XYEval: Agents say yes to bad advicecited by 0
3 more
  1. 2026Learning When and How to Intervene: A Hindsight-Distilled Sentinel for Coding Agentscited by 0
  2. 2026What Can Component-Replacement Evidence Establish? A Critical Scoping Review of Local Decisions in LLM Agentscited by 0
  3. 2026TestJack: Should you trust the results in coding benchmarks? Agentic Coding Benchmarks Auditing via Evaluator Evolutioncited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Uncertainty-Aware Clarification in LLM Agents with Information Gain
  2. 2026Uncertainty Decomposition for Clarification Seeking in LLM Agents
  3. 2025Can Code Language Models Learn Clarification-Seeking Behaviors?
  4. 2026AgentRadio: Passive Awareness for Long-Horizon Multi-Agent Collaboration
  5. 2025Structured Uncertainty guided Clarification for LLM Agents
  6. 2026From Question Answering to Task Completion: A Survey on Agent System and Harness Design
  7. 2026CONTRA: Discovering and Qualifying Behavior-Changing Questions for Selective Clarification in LLM Code Generation
  8. 2026Uncertainty Quantification in LLM Agents: Foundations, Emerging Challenges, and Opportunities