# Don't Start What You Can't Finish: A Counterfactual Audit of Support-State Triage in LLM Agents

arXiv: https://arxiv.org/abs/2604.16752 · HTML: https://mlsanity.com/paper/2604.16752/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Placed in Software engineering coding agents, one of 16 directions in LLM agents and tool use.
- Builds on 23 of its 28 references that are in the map.
- First posted on arXiv 17 Apr 2026.

## Placed in

- LLM agents and tool use / [Software engineering coding agents](https://mlsanity.com/agents/software-engineering-coding-agents/)

## Builds on (most cited first, 23 of 23)

- [Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena](https://mlsanity.com/paper/2306.05685/) (2023)
- [Language Models (Mostly) Know What They Know](https://mlsanity.com/paper/2207.05221/) (2022)
- [Can LLMs Express Their Uncertainty? An Empirical Evaluation of Confidence Elicitation in LLMs](https://mlsanity.com/paper/2306.13063/) (2023)
- [Learning the Difference that Makes a Difference with Counterfactually-Augmented Data](https://mlsanity.com/paper/1909.12434/) (2019)
- [AbstentionBench: Reasoning LLMs Fail on Unanswerable Questions](https://mlsanity.com/paper/2506.09038/) (2025)
- [T-Eval: Evaluating the Tool Utilization Capability of Large Language Models Step by Step](https://mlsanity.com/paper/2312.14033/) (2023)
- [QuestBench: Can LLMs ask the right question to acquire information in reasoning tasks?](https://mlsanity.com/paper/2503.22674/) (2025)
- [Do LLMs Know When to NOT Answer? Investigating Abstention Abilities of Large Language Models](https://mlsanity.com/paper/2407.16221/) (2024)
- [Structured Uncertainty guided Clarification for LLM Agents](https://mlsanity.com/paper/2511.08798/) (2025)
- [HiL-Bench (Human-in-Loop Benchmark): Do Agents Know When to Ask for Help?](https://mlsanity.com/paper/2604.09408/) (2026)
- [Ask or Assume? Uncertainty-Aware Clarification-Seeking in Coding Agents](https://mlsanity.com/paper/2603.26233/) (2026)
- [When and What to Ask: AskBench and Rubric-Guided RLVR for LLM Clarification](https://mlsanity.com/paper/2602.11199/) (2026)
- [Saving SWE-Bench: A Benchmark Mutation Approach for Realistic Agent Evaluation](https://mlsanity.com/paper/2510.08996/) (2025)
- [Accurate Failure Prediction in Agents Does Not Imply Effective Failure Prevention](https://mlsanity.com/paper/2602.03338/) (2026)
- [Over-Searching in Search-Augmented Large Language Models](https://mlsanity.com/paper/2601.05503/) (2026)
- [Efficient Benchmarking of AI Agents](https://mlsanity.com/paper/2603.23749/) (2026)
- [RefusalBench: Generative Evaluation of Selective Refusal in Grounded Language Models](https://mlsanity.com/paper/2510.10390/) (2025)
- [CAR-bench: Evaluating the Consistency and Limit-Awareness of LLM Agents under Real-World Uncertainty](https://mlsanity.com/paper/2601.22027/) (2026)
- [MASH: Modeling Abstention via Selective Help-Seeking](https://mlsanity.com/paper/2510.01152/) (2025)
- [LHAW: Controllable Underspecification for Long-Horizon Tasks](https://mlsanity.com/paper/2602.10525/) (2026)
- [I-CALM: Incentivizing Confidence-Aware Abstention for LLM Selective Answering](https://mlsanity.com/paper/2604.03904/) (2026)
- [Is Your Automated Software Engineer Trustworthy?](https://mlsanity.com/paper/2506.17812/) (2025)
- [AWARE-US: Preference-Aware Infeasibility Resolution in Tool-Calling Agents](https://mlsanity.com/paper/2601.02643/) (2026)

## Abstract (arXiv)

Current agent evaluations largely reward execution on fully specified tasks, while recent work studies clarification [11, 22, 2], capability awareness [9, 1], abstention [8, 14], and search termination [20, 5] mostly in isolation. This leaves open whether agents can diagnose why a task is blocked before acting. We introduce the Support-State Triage Audit (SSTA-32), a matched-item diagnostic framework in which minimal counterfactual edits flip the same base request across four support states: Complete (ANSWER), Clarifiable (CLARIFY), Support-Blocked (REQUEST SUPPORT), and Unsupported-Now (ABSTAIN). We evaluate a frontier model under four prompting conditions - Direct, Action-Only, Confidence-Only, and a typed Preflight Support Check (PSC) - using Dual-Persona Auto-Auditing (DPAA) with deterministic heuristic scoring. Default execution overcommits heavily on non-complete tasks (41.7% overcommitment rate). Scalar confidence mapping avoids overcommitment but collapses the three-way deferral space (58.3% typed deferral accuracy). Conversely, both Action-Only and PSC achieve 91.7% typed deferral accuracy by surfacing the categorical ontology in the prompt. Targeted ablations confirm that removing the support-sufficiency dimension selectively degrades REQUEST SUPPORT accuracy, while removing the evidence-sufficiency dimension triggers systematic overcommitment on unsupported items. Because DPAA operates within a single context window, these results represent upper-bound capability estimates; nonetheless, the structural findings indicate that frontier models possess strong latent triage capabilities that require explicit categorical decision paths to activate safely.