# AM-Bench: Understanding Agents under Task Underspecification across Models and Harnesses

arXiv: https://arxiv.org/abs/2610.18090 · HTML: https://mlsanity.com/paper/2610.18090/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Placed in General LLM agent frameworks and benchmarks, one of 16 directions in LLM agents and tool use.
- Builds on 19 of its 22 references that are in the map.
- First posted on arXiv 13 Oct 2026.

## Placed in

- LLM agents and tool use / [General LLM agent frameworks and benchmarks](https://mlsanity.com/agents/general-llm-agent-frameworks-and-benchmarks/)

## Builds on (most cited first, 19 of 19)

- [WebArena: A Realistic Web Environment for Building Autonomous Agents](https://mlsanity.com/paper/2307.13854/) (2023)
- [ToolLLM: Facilitating Large Language Models to Master 16000+ Real-world APIs](https://mlsanity.com/paper/2307.16789/) (2023)
- [τ-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domains](https://mlsanity.com/paper/2406.12045/) (2024)
- [Terminal-Bench: Benchmarking Agents on Hard, Realistic Tasks in Command Line Interfaces](https://mlsanity.com/paper/2601.11868/) (2026)
- [Identifying the Risks of LM Agents with an LM-Emulated Sandbox](https://mlsanity.com/paper/2309.15817/) (2023)
- [InjecAgent: Benchmarking Indirect Prompt Injections in Tool-Integrated Large Language Model Agents](https://mlsanity.com/paper/2403.02691/) (2024)
- [AgentHarm: A Benchmark for Measuring Harmfulness of LLM Agents](https://mlsanity.com/paper/2410.09024/) (2024)
- [AppWorld: A Controllable World of Apps and People for Benchmarking Interactive Coding Agents](https://mlsanity.com/paper/2407.18901/) (2024)
- [MINT: Evaluating LLMs in Multi-turn Interaction with Tools and Language Feedback](https://mlsanity.com/paper/2309.10691/) (2023)
- [WASP: Benchmarking Web Agent Security Against Prompt Injection Attacks](https://mlsanity.com/paper/2504.18575/) (2025)
- [Harness-Bench: Measuring Harness Effects across Models in Realistic Agent Workflows](https://mlsanity.com/paper/2605.27922/) (2026)
- [CLAMBER: A Benchmark of Identifying and Clarifying Ambiguous Information Needs in Large Language Models](https://mlsanity.com/paper/2405.12063/) (2024)
- [Learning to Ask: When LLM Agents Meet Unclear Instruction](https://mlsanity.com/paper/2409.00557/) (2024)
- [Structured Uncertainty guided Clarification for LLM Agents](https://mlsanity.com/paper/2511.08798/) (2025)
- [Asking Clarification Questions to Handle Ambiguity in Open-Domain QA](https://mlsanity.com/paper/2305.13808/) (2023)
- [Coding Agents Are Guessing: Measuring Action-Boundary Violations in Underspecified DevOps Instructions](https://mlsanity.com/paper/2607.02294/) (2026)
- [AppWorld-UL: Benchmarking Diverse Agent-User Interactions for Tool-Use](https://mlsanity.com/paper/2607.20536/) (2026)
- [HarnessSafe: Evaluating Safety Across Persistent Carriers in Agent Harnesses](https://mlsanity.com/paper/2608.06984/) (2026)
- [HarnessRisk: A Lifecycle-Oriented Benchmark for Agent Harness Safety](https://mlsanity.com/paper/2608.17597/) (2026)

## Abstract (arXiv)

Ambiguous or incomplete task specifications complicate both execution and clarification for language-model agents. Yet agents' responses to these information gaps remain poorly characterized across models and execution harnesses. We present AM-Bench, a benchmark built from 48 tasks adapted from Terminal-Bench 2.0 and Harness-Bench. Each task includes a fully specified reference and six variants spanning ambiguous or missing goals, context, and constraints. We evaluate every pairing of five models and four harnesses over 20,160 runs. Our results reveal distinct execution deficits: goal gaps hinder completion, context gaps increase token use, and constraint gaps weaken compliance without necessarily preventing completion. Missingness lowers success more than ambiguity in all three domains. These effects also reshape system comparisons: advantages under complete information need not persist, and rankings depend on model × harness pairings and the evaluation criterion -- completion, success, or repeat consistency. Estimates based on observed clarification requests and matched fully specified runs further suggest that clarification could recover unfinished work while leaving most constraint violations unresolved. Together, these findings motivate joint model-harness evaluation across information conditions and targeted clarification of unresolved requirements before dependent actions. To support future research on agents under task underspecification, we provide the AM-Bench project repository at https://github.com/s4-lab-cuhksz/AM-Bench.git.