mlsanityData to 7 Oct 2026

Where MLLMs Fail and Why: Causal Task Decomposition for Capability Failure Diagnosis

arXiv 2609.38851 ↗First posted 30 Sep 2026cs.CL cs.AI cs.CVcited by 0

Not on the map. Why?

Abstract (arXiv)

End-to-end accuracy on compositional tasks records how often MLLMs fail, but cannot distinguish whether a failure reflects an intrinsic deficit in the targeted capability or a cascading error from an upstream prerequisite. We propose a causal decomposition framework that isolates these two failure modes through controlled interventions on the prerequisite dependencies of each task. Our capability metrics (NC, IC, RC) score each task under unassisted, correct, or incorrect prerequisites to diagnose where failures arise; contribution metrics (N-Score, S-Score), adapted from probabilities of causation, quantify each prerequisite's necessity and sufficiency to determine why. We instantiate the framework in CADET, a diagnostic benchmark of 10 composite tasks decomposed into 46 unit tasks with over 33,000 human-annotated questions spanning perception, spatial, temporal, and cognitive categories. Diagnosing frontier MLLMs with our framework uncovers systematic patterns that end-to-end accuracy obscures. Capability-wise, supplying correct prerequisites eliminates 54% of errors on cognitive tasks, lifting them from weakest to above spatial and temporal. Prerequisite-wise, causal contributions are concentrated in a few critical prerequisites, and supplying the single most important one alone captures 84% of the gain from supplying all prerequisites.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 37 of its 50 references that are in the map. First posted on arXiv 30 Sep 2026.

Builds on

37 of 50 references.

References · 37

  1. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  2. 2023Let's Verify Step by Stepcited by 3,598
  3. 2022Locating and Editing Factual Associations in GPTcited by 2,609
  4. 2023MMBench: Is Your Multi-modal Model an All-around Player?cited by 2,105
  5. 2023MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGIcited by 2,044
  6. 2023MathVista: Evaluating Mathematical Reasoning of Foundation Models in Visual Contextscited by 1,582
  7. 2023Language Models Don't Always Say What They Think: Unfaithful Explanations in Chain-of-Thought Promptingcited by 1,081
  8. 2022Measuring and Narrowing the Compositionality Gap in Language Modelscited by 1,054
  9. 2024Eyes Wide Shut? Exploring the Visual Shortcomings of Multimodal LLMscited by 719
  10. 2024Thinking in Space: How Multimodal Large Language Models See, Remember, and Recall Spacescited by 580
  11. 2024MathVerse: Does Your Multi-modal LLM Truly See the Diagrams in Visual Math Problems?cited by 573
  12. 2023HallusionBench: An Advanced Diagnostic Suite for Entangled Language Hallucination and Visual Illusion in Large Vision-Language Modelscited by 572
25 more
  1. 2024BLINK: Multimodal Large Language Models Can See but Not Perceivecited by 529
  2. 2026Kimi K2.5: Visual Agentic Intelligencecited by 449
  3. 2025Reasoning Models Don't Always Say What They Thinkcited by 388
  4. 2024Visual CoT: Advancing Multi-Modal Language Models with a Comprehensive Dataset and Benchmark for Chain-of-Thought Reasoningcited by 267
  5. 2025Which Agent Causes Task Failures and When? On Automated Failure Attribution of LLM Multi-Agent Systemscited by 170
  6. 2025Can MLLMs Reason in Multimodality? EMMA: An Enhanced MultiModal ReAsoning Benchmarkcited by 132
  7. 2023Interpretability at Scale: Identifying Causal Mechanisms in Alpacacited by 102
  8. 2013Probabilities of Causation: Bounds and Identificationcited by 89
  9. 2024FaithEval: Can Your Language Model Stay Faithful to Context, Even If "The Moon is Made of Marshmallows"cited by 85
  10. 2013Counterfactual Probabilities: Computational Methods, Bounds and Applicationscited by 67
  11. 2021Explaining Black-Box Algorithms Using Probabilistic Contrastive Counterfactualscited by 55
  12. 2024Is Your Model Really A Good Math Reasoner? Evaluating Mathematical Reasoning with Checklistcited by 48
  13. 2021Local Explanations via Necessity and Sufficiency: Unifying Theory and Practicecited by 45
  14. 2024MEGA-Bench: Scaling Multimodal Evaluation to over 500 Real-World Taskscited by 41
  15. 2026BabyVision: Visual Reasoning Beyond Languagecited by 39
  16. 2025Treble Counterfactual VLMs: A Causal Approach to Hallucinationcited by 33
  17. 2025DoVer: Intervention-Driven Auto Debugging for LLM Multi-Agent Systemscited by 23
  18. 2024ROADWork: A Dataset and Benchmark for Learning to Recognize, Observe, Analyze and Drive Through Work Zonescited by 20
  19. 2026Causal Agent Replay: Counterfactual Attribution for LLM-Agent Failurescited by 18
  20. 2026From Flat Logs to Causal Graphs: Hierarchical Failure Attribution for LLM-based Multi-Agent Systemscited by 17
  21. 2026CausalFlow: Causal Attribution and Counterfactual Repair for LLM Agent Failurescited by 17
  22. 2025Causal Sufficiency and Necessity Improves Chain-of-Thought Reasoningcited by 16
  23. 2024Does Reasoning Emerge? Examining the Probabilities of Causation in Large Language Modelscited by 8
  24. 2025Reasoning is All You Need for Video Generalization: A Counterfactual Benchmark with Sub-question Evaluationcited by 6
  25. 2026Which Modality Decides? Counterfactual Modality Attribution for Multimodal LLMscited by 1

Built on by

Not cited yet.