Implicit Intelligence -- Evaluating Agents on What Users Don't Say
On the map: LLM agents and tool use → General LLM agent frameworks and benchmarks
Abstract (arXiv)
Real-world requests to AI agents are fundamentally underspecified. Natural human communication relies on shared context and unstated constraints that speakers expect listeners to infer. Current agentic benchmarks test explicit instruction-following but fail to evaluate whether agents can reason about implicit requirements spanning accessibility needs, privacy boundaries, catastrophic risks, and contextual constraints. We present Implicit Intelligence, an evaluation framework testing whether AI agents can move beyond prompt-following to become genuine goal-fulfillers, paired with Agent-as-a-World (AaW), a harness where interactive worlds are defined in human-readable YAML files and simulated by language models. Our scenarios feature apparent simplicity in user requests, hidden complexity in correct solutions, and discoverability of constraints through environmental exploration. Evaluating 16 frontier and open-weight models across 205 scenarios, we find that even the best-performing model achieves only 48.3% scenario pass rate, revealing substantial room for improvement in bridging the gap between literal instruction-following and human-like contextual reasoning.
Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in General LLM agent frameworks and benchmarks, one of 16 directions in LLM agents and tool use. Builds on 13 of its 32 references that are in the map. First posted on arXiv 23 Feb 2026.
Builds on
13 of 32 references.
Closest ancestors · 2
Same direction, newest first
- 2025General Agentic Planning Through Simulative Reasoning with World Models31 Jul
- 2024τ-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domains17 Jun
Other references · 11
- 2023SWE-bench: Can Language Models Resolve Real-World GitHub Issues?cited by 3,193
- 2023ToolLLM: Facilitating Large Language Models to Master 16000+ Real-world APIscited by 1,880
- 2022WebShop: Towards Scalable Real-World Web Interaction with Grounded Language Agentscited by 1,197
- 2020ALFWorld: Aligning Text and Embodied Environments for Interactive Learningcited by 1,143
- 2024Scaling Synthetic Data Creation with 1,000,000,000 Personascited by 397
- 2022The Alignment Problem from a Deep Learning Perspectivecited by 246
- 2025Simulating Environments with Reasoning Models for Agent Trainingcited by 52
- 2025Implicit Reasoning in Large Language Models: A Comprehensive Surveycited by 42
- 2025MI9: An Integrated Runtime Governance Framework for Agentic AIcited by 18
- 2025EIFBENCH: Extremely Complex Instruction Following Benchmark for Large Language Modelscited by 9
- 2025AURA: An Agent Autonomy Risk Assessment Frameworkcited by 2
Built on by
Not cited yet.
Similar, not cited
Close in wording, no citation either way
- 2024Infer Human's Intentions Before Following Natural Language Instructions
- 2024Tell Me More! Towards Implicit User Intention Understanding of Language Model Driven Agents
- 2024AgentSense: Benchmarking Social Intelligence of Language Agents through Interactive Scenarios
- 2025Emergence: Overcoming Privileged Information Bias in Asymmetric Embodied Agents via Active Querying
- 2025Why do AI agents communicate in human language?
- 2026FORTIS: Benchmarking Over-Privilege in Agent Skills
- 2026ACCORD: Action-Conditioned Contextual Grounding for Language Agents
- 2023Asking Before Acting: Gather Information in Embodied Decision Making with Language Models