CARD: Diagnosing Belief to Action Routing Failures in Vision Language Models
On the map: Multimodal LLMs / vision-language models → Embodied and 3D vision-language models
also in 1 more
Abstract (arXiv)
Linear probes and activation steering have uncovered that vision-language models (VLMs) internally represent mental states such as agents' beliefs, knowledge, and intentions. However, it is unclear whether and how these representations are used by downstream predictions along these axes. To close this gap, we introduce Cross-Axis Routing Diagnostic (CARD), which steers activations along one axis while measuring the response of a different axis's prediction. Applied to open-weight VLMs on Relay Chain -- a new cooperative grid-world benchmark we propose -- we diagnose a critical routing failure: models fail to incorporate belief representations into their next action prediction, effectively leaving valuable information about their partners unused.
Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Embodied and 3D vision-language models, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 21 of its 39 references that are in the map. First posted on arXiv 21 Aug 2026.
Builds on
21 of 39 references.
References · 21
- 2022Locating and Editing Factual Associations in GPTcited by 2,691
- 2016CLEVR: A Diagnostic Dataset for Compositional Language and Elementary Visual Reasoningcited by 1,457
- 2016Understanding intermediate layers using linear classifier probescited by 1,353
- 2023Inference-Time Intervention: Eliciting Truthful Answers from a Language Modelcited by 1,302
- 2022Interpretability in the Wild: a Circuit for Indirect Object Identification in GPT-2 smallcited by 1,165
- 2023Steering Llama 2 via Contrastive Activation Additioncited by 1,085
- 2021Probing Classifiers: Promises, Shortcomings, and Advancescited by 876
- 2019Designing and Interpreting Probes with Control Taskscited by 640
- 2023Dissecting Recall of Factual Associations in Auto-Regressive Language Modelscited by 528
- 2021Causal Abstractions of Neural Networkscited by 422
- 2025SmolVLM: Redefining small and efficient multimodal modelscited by 237
- 2023Linearity of Relation Decoding in Transformer Language Modelscited by 200
9 more
- 2023FANToM: A Benchmark for Stress-testing Machine Theory of Mind in Interactionscited by 171
- 2024MMToM-QA: Multimodal Theory of Mind Question Answeringcited by 85
- 2024ToMBench: Benchmarking Theory of Mind in Large Language Modelscited by 72
- 2024SimpleToM: Exposing the Gap between Explicit ToM Inference and Implicit ToM Application in LLMscited by 57
- 2024MuMA-ToM: Multi-modal Multi-Agent Theory of Mindcited by 51
- 2024Brittle Minds, Fixable Activations: Understanding Belief Representations in Language Modelscited by 8
- 2026Circuit Tracing in Vision-Language Models: Understanding the Internal Mechanisms of Multimodal Thinkingcited by 8
- 2025ToM-SSI: Evaluating Theory of Mind in Situated Social Interactionscited by 5
- 2025MindPower: Enabling Theory-of-Mind Reasoning in VLM-based Embodied Agentscited by 5
Built on by
Not cited yet.
Similar, not cited
Close in wording, no citation either way
- 2025DeepThinkVLA: Enhancing Reasoning Capability of Vision-Language-Action Models
- 2025TRACE: A Self-Improving Framework for Robot Behavior Forecasting with Vision-Language Models
- 2026Your Vision-Language-Action Model Already Has Attention Heads For Path Deviation Detection
- 2025Do What You Say: Steering Vision-Language-Action Models via Runtime Reasoning-Action Alignment Verification
- 2025Mechanistic interpretability for steering vision-language-action models
- 2026DualCoT-VLA: Visual-Linguistic Chain of Thought via Parallel Reasoning for Vision-Language-Action Models
- 2026VLA-Trace: Diagnosing Vision-Language-Action Models through Representation and Behavior Tracing
- 2026DiaVLo: Diagnosing Behaviours of Vision-Language Models