CockpitHAT: Dependency-Graph-Driven Hierarchical Attribution for Embodied Multi-Agent Cockpits
On the map: LLM agents and tool use → Agent threats, guardrails and governance
Abstract (arXiv)
LLM multi-agent systems suffer from Correctness Collapse, where high task-level accuracy conceals severe process-level failures. This is especially hazardous in safety-critical embodied settings such as automotive cockpits, where lexically correct utterances may trigger dangerous physical operations. Existing attribution methods rely on text traces alone, missing dependency structure, multi-channel evidence, and safety-aware evaluation. We introduce CockpitHAT, a hierarchical attribution framework that replaces positional windows with dependency-distance thresholds from interaction DAGs, integrates multi-channel evidence via an embodied adapter, and applies a safety-uplift to high-risk failures during confidence-weighted analyst consensus. We further release CockpitBench, a benchmark of 212 annotated failure traces spanning dialogue, vehicle-state, environmental, and memory channels, each labeled with ISO 26262 ASIL severity via three-expert consensus. On the public Who&When benchmark, CockpitHAT achieves agent-level / step-exact accuracies of 77.9% / 37.8% on the Hand-Crafted split and 86.5% / 46.0% on the Algorithm-Generated split, surpassing the text-only SOTA ECHO by up to 17.6 / 16.7 points. On CockpitBench, it attains 78.3% agent-level and 38.2% step-exact accuracy. These results establish dependency-aware, multi-channel, risk-calibrated attribution as an effective paradigm for reliable failure diagnosis in real-world embodied LLM multi-agent systems.
Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Placed in Agent threats, guardrails and governance, one of 16 directions in LLM agents and tool use. Builds on 22 of its 41 references that are in the map. First posted on arXiv 3 Aug 2026.
Builds on
22 of 41 references.
Closest ancestors · 1
Same direction, newest first
- 2025IS-Bench: Evaluating Interactive Safety of VLM-Driven Embodied Agents in Daily Household Tasks19 Jun
Other references · 21
- 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
- 2023Reflexion: Language Agents with Verbal Reinforcement Learningcited by 4,276
- 2023Toolformer: Language Models Can Teach Themselves to Use Toolscited by 4,182
- 2023SWE-bench: Can Language Models Resolve Real-World GitHub Issues?cited by 3,373
- 2023WebArena: A Realistic Web Environment for Building Autonomous Agentscited by 2,006
- 2023AutoGen: Enabling Next-Gen LLM Applications via Multi-Agent Conversationcited by 1,988
- 2023Gorilla: Large Language Model Connected with Massive APIscited by 1,332
- 2023FActScore: Fine-grained Atomic Evaluation of Factual Precision in Long Form Text Generationcited by 1,282
- 2023Teaching Large Language Models to Self-Debugcited by 988
- 2023ChatDev: Communicative Agents for Software Developmentcited by 919
- 2022Solving math word problems with process- and outcome-based feedbackcited by 901
- 2024From Crowdsourced Data to High-Quality Benchmarks: Arena-Hard and BenchBuilder Pipelinecited by 500
9 more
- 2025EmbodiedBench: Comprehensive Benchmarking Multi-modal Large Language Models for Vision-Driven Embodied Agentscited by 221
- 2025MultiAgentBench: Evaluating the Collaboration and Competition of LLM agentscited by 178
- 2019Talk2Car: Taking Control of Your Self-Driving Carcited by 142
- 2025Who is Introducing the Failure? Automatically Attributing Failures of Multi-Agent Systems via Spectrum Analysiscited by 33
- 2025Where Did It All Go Wrong? A Hierarchical Look into Multi-Agent Error Attributioncited by 27
- 2026Rethinking Failure Attribution in Multi-Agent Systems: A Multi-Perspective Benchmark and Evaluationcited by 24
- 2026AgentDrive: An Open Benchmark Dataset for Agentic AI Reasoning with LLM-Generated Scenarios in Autonomous Systemscited by 4
- 2025ChatGPT on the Road: Leveraging Large Language Model-Powered In-vehicle Conversational Agents for Safer and More Enjoyable Driving Experiencecited by 3
- 2025Benchmarking Contextual Understanding for In-Car Conversational Systemscited by 2
Built on by
Citing papers per year: 2026: 1.
1 paper.
Similar, not cited
Close in wording, no citation either way
- 2024SafeAgentBench: A Benchmark for Safe Task Planning of Embodied LLM Agents
- 2026DCFA: Dual-view Causal-inspired Attribution for Failure Reasoning in LLM-based Multi-agent Systems
- 2026Seeing the Whole Elephant: A Benchmark for Failure Attribution in LLM-based Multi-Agent Systems
- 2026PilotBench: A Benchmark for General Aviation Agents with Safety Constraints
- 2026VerifyMAS: Hypothesis Verification for Failure Attribution in LLM Multi-Agent Systems
- 2026AgentBoundary: Counterfactual Evaluation of Safety in Tool-Using LLM Agents
- 2026Adversarial Pragmatics for AI Safety Evaluation: A Diagnostic Framework and Seed Benchmark for Language-Mediated Control
- 2026StepFinder: A Temporal Semantic Framework for Failure Attribution in Multi-Agent Systems