# CAR-bench: Evaluating the Consistency and Limit-Awareness of LLM Agents under Real-World Uncertainty

arXiv: https://arxiv.org/abs/2601.22027 · HTML: https://mlsanity.com/paper/2601.22027/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- Placed in General LLM agent frameworks and benchmarks, one of 16 directions in LLM agents and tool use.
- Builds on 21 of its 50 references that are in the map.
- First posted on arXiv 29 Jan 2026.

## Placed in

- LLM agents and tool use / [General LLM agent frameworks and benchmarks](https://mlsanity.com/agents/general-llm-agent-frameworks-and-benchmarks/)

## Builds on (most cited first, 21 of 21)

- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [ReAct: Synergizing Reasoning and Acting in Language Models](https://mlsanity.com/paper/2210.03629/) (2022)
- [Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena](https://mlsanity.com/paper/2306.05685/) (2023)
- [Generative Agents: Interactive Simulacra of Human Behavior](https://mlsanity.com/paper/2304.03442/) (2023)
- [SWE-bench: Can Language Models Resolve Real-World GitHub Issues?](https://mlsanity.com/paper/2310.06770/) (2023)
- [A Survey on Large Language Model based Autonomous Agents](https://mlsanity.com/paper/2308.11432/) (2023)
- [WebArena: A Realistic Web Environment for Building Autonomous Agents](https://mlsanity.com/paper/2307.13854/) (2023)
- [ToolLLM: Facilitating Large Language Models to Master 16000+ Real-world APIs](https://mlsanity.com/paper/2307.16789/) (2023)
- [τ-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domains](https://mlsanity.com/paper/2406.12045/) (2024)
- [WebShop: Towards Scalable Real-World Web Interaction with Grounded Language Agents](https://mlsanity.com/paper/2207.01206/) (2022)
- [ALFWorld: Aligning Text and Embodied Environments for Interactive Learning](https://mlsanity.com/paper/2010.03768/) (2020)
- [MultiWOZ -- A Large-Scale Multi-Domain Wizard-of-Oz Dataset for Task-Oriented Dialogue Modelling](https://mlsanity.com/paper/1810.00278/) (2018)
- [ToolSandbox: A Stateful, Conversational, Interactive Evaluation Benchmark for LLM Tool Use Capabilities](https://mlsanity.com/paper/2408.04682/) (2024)
- [MetaTool Benchmark for Large Language Models: Deciding Whether to Use Tools and Which to Use](https://mlsanity.com/paper/2310.03128/) (2023)
- [APIGen-MT: Agentic Pipeline for Multi-Turn Data Generation via Simulated Agent-Human Interplay](https://mlsanity.com/paper/2504.03601/) (2025)
- [Planning, Creation, Usage: Benchmarking LLMs for Comprehensive Tool Utilization in Real-World Complex Scenarios](https://mlsanity.com/paper/2401.17167/) (2024)
- [Action-Based Conversations Dataset: A Corpus for Building More In-Depth Task-Oriented Dialogue Systems](https://mlsanity.com/paper/2104.00783/) (2021)
- [PlatoLM: Teaching LLMs in Multi-Round Dialogue via a User Simulator](https://mlsanity.com/paper/2308.11534/) (2023)
- [CarMem: Enhancing Long-Term Memory in LLM Voice Assistants through Category-Bounding](https://mlsanity.com/paper/2501.09645/) (2025)
- [Beyond Static Evaluation: A Dynamic Approach to Assessing AI Assistants' API Invocation Capabilities](https://mlsanity.com/paper/2403.11128/) (2024)
- [Integrating Generative Artificial Intelligence in Intelligent Vehicle Systems](https://mlsanity.com/paper/2305.17137/) (2023)

## Built on by (most cited first, 8 of 8)

- [Agent2Agent Threats in Safety-Critical LLM Assistants: A Human-Centric Taxonomy](https://mlsanity.com/paper/2602.05877/) (2026)
- [TRACE: A Self-Evolving Skill Bank for Consistent, Limit-Aware LLM Agents](https://mlsanity.com/paper/2608.22793/) (2026)
- [Don't Make Models Guess Security and Safety: Symbolic Guardrails for Domain-Specific AI Agents](https://mlsanity.com/paper/2604.15579/) (2026)
- [What Are You Doing? Effects of Intermediate Feedback from Agentic LLM In-Car Assistants During Multi-Step Processing](https://mlsanity.com/paper/2602.15569/) (2026)
- [Don't Start What You Can't Finish: A Counterfactual Audit of Support-State Triage in LLM Agents](https://mlsanity.com/paper/2604.16752/) (2026)
- [From Natural Language Policies to Executable Obligations: A Verification Harness for Dependable In-Car LLM Agents](https://mlsanity.com/paper/2608.23282/) (2026)
- [From Intent to Action: Benchmarking LLM Safety in Vehicle Voice Command Authorization](https://mlsanity.com/paper/2609.19630/) (2026)
- [Policy as Code: A Coroutine-Bridge Harness for Fast-Reasoning Reliability on CAR-bench](https://mlsanity.com/paper/2609.29251/) (2026)

## Abstract (arXiv)

Existing benchmarks for Large Language Model (LLM) agents focus on task completion under idealistic settings but overlook reliability in real-world, user-facing applications. In domains, such as in-car voice assistants, users often issue incomplete or ambiguous requests, creating intrinsic uncertainty that agents must manage through dialogue, tool use, and policy adherence. We introduce CAR-bench, a benchmark for evaluating consistency, uncertainty handling, and capability awareness in multi-turn, tool-using LLM agents in an in-car assistant domain. The environment features an LLM-simulated user, domain policies, and 58 interconnected tools spanning navigation, productivity, charging, and vehicle control. Beyond standard task completion, CAR-bench introduces Hallucination tasks that test agents' limit-awareness under missing tools or information, and Disambiguation tasks that require resolving uncertainty through clarification or internal information gathering. Baseline results reveal large gaps between occasional and consistent success on all task types. Even frontier reasoning LLMs achieve less than 50% consistent pass rate on Disambiguation tasks due to premature actions, and frequently violate policies or fabricate information to satisfy user requests in Hallucination tasks, underscoring the need for more reliable and self-aware LLM agents in real-world settings.