mlsanityData to 7 Oct 2026

Don't Start What You Can't Finish: A Counterfactual Audit of Support-State Triage in LLM Agents

arXiv 2604.16752 ↗First posted 17 Apr 2026cs.AIcited by 0

On the map: LLM agents and tool use → Software engineering coding agents

Abstract (arXiv)

Current agent evaluations largely reward execution on fully specified tasks, while recent work studies clarification [11, 22, 2], capability awareness [9, 1], abstention [8, 14], and search termination [20, 5] mostly in isolation. This leaves open whether agents can diagnose why a task is blocked before acting. We introduce the Support-State Triage Audit (SSTA-32), a matched-item diagnostic framework in which minimal counterfactual edits flip the same base request across four support states: Complete (ANSWER), Clarifiable (CLARIFY), Support-Blocked (REQUEST SUPPORT), and Unsupported-Now (ABSTAIN). We evaluate a frontier model under four prompting conditions - Direct, Action-Only, Confidence-Only, and a typed Preflight Support Check (PSC) - using Dual-Persona Auto-Auditing (DPAA) with deterministic heuristic scoring. Default execution overcommits heavily on non-complete tasks (41.7% overcommitment rate). Scalar confidence mapping avoids overcommitment but collapses the three-way deferral space (58.3% typed deferral accuracy). Conversely, both Action-Only and PSC achieve 91.7% typed deferral accuracy by surfacing the categorical ontology in the prompt. Targeted ablations confirm that removing the support-sufficiency dimension selectively degrades REQUEST SUPPORT accuracy, while removing the evidence-sufficiency dimension triggers systematic overcommitment on unsupported items. Because DPAA operates within a single context window, these results represent upper-bound capability estimates; nonetheless, the structural findings indicate that frontier models possess strong latent triage capabilities that require explicit categorical decision paths to activate safely.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Software engineering coding agents, one of 16 directions in LLM agents and tool use. Builds on 23 of its 28 references that are in the map. First posted on arXiv 17 Apr 2026.

Builds on

23 of 28 references.

Closest ancestors · 2

Same direction, newest first

  1. 2026Accurate Failure Prediction in Agents Does Not Imply Effective Failure Prevention3 Feb
  2. 2025Saving SWE-Bench: A Benchmark Mutation Approach for Realistic Agent Evaluation10 Oct

Other references · 21

  1. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  2. 2022Language Models (Mostly) Know What They Knowcited by 1,783
  3. 2023Can LLMs Express Their Uncertainty? An Empirical Evaluation of Confidence Elicitation in LLMscited by 946
  4. 2019Learning the Difference that Makes a Difference with Counterfactually-Augmented Datacited by 428
  5. 2025AbstentionBench: Reasoning LLMs Fail on Unanswerable Questionscited by 129
  6. 2023T-Eval: Evaluating the Tool Utilization Capability of Large Language Models Step by Stepcited by 91
  7. 2025QuestBench: Can LLMs ask the right question to acquire information in reasoning tasks?cited by 56
  8. 2024Do LLMs Know When to NOT Answer? Investigating Abstention Abilities of Large Language Modelscited by 55
  9. 2025Structured Uncertainty guided Clarification for LLM Agentscited by 34
  10. 2026HiL-Bench (Human-in-Loop Benchmark): Do Agents Know When to Ask for Help?cited by 16
  11. 2026Ask or Assume? Uncertainty-Aware Clarification-Seeking in Coding Agentscited by 15
  12. 2026When and What to Ask: AskBench and Rubric-Guided RLVR for LLM Clarificationcited by 12
9 more
  1. 2026Over-Searching in Search-Augmented Large Language Modelscited by 10
  2. 2026Efficient Benchmarking of AI Agentscited by 9
  3. 2025RefusalBench: Generative Evaluation of Selective Refusal in Grounded Language Modelscited by 8
  4. 2026CAR-bench: Evaluating the Consistency and Limit-Awareness of LLM Agents under Real-World Uncertaintycited by 8
  5. 2026LHAW: Controllable Underspecification for Long-Horizon Taskscited by 5
  6. 2026I-CALM: Incentivizing Confidence-Aware Abstention for LLM Selective Answeringcited by 4
  7. 2025Is Your Automated Software Engineer Trustworthy?cited by 3
  8. 2025MASH: Modeling Abstention via Selective Help-Seekingcited by 3
  9. 2026AWARE-US: Preference-Aware Infeasibility Resolution in Tool-Calling Agentscited by 1

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026Quantifying Overclaiming Propensity in Frontier LLM Agents
  2. 2026BRA-Audit: Budgeted Runtime Auditing for LLM Multi-Agent Systems via Cumulative-Exposure Audit-Point Placement
  3. 2026StateWise: Diagnosing and Repairing Persistent Operational State Before Agent Actions
  4. 2026Beyond Task Completion: Revealing Corrupt Success in LLM Agents through Procedure-Aware Evaluation
  5. 2026Insights Generator: Systematic Corpus-Level Trace Diagnostics for LLM Agents
  6. 2026Auditing Action Settlement in LLM Agent Environments: Order, Progress, and Replay
  7. 2025Traceability and Accountability in Role-Specialized Multi-Agent LLM Pipelines
  8. 2026Early Diagnosis of Wasted Computation in Multi-Agent LLM Systems via Failure-Aware Observability