mlsanityData to 14 Oct 2026

EDCT-Bench: Uncovering Faithfulness Gaps in VLMs via Explanation-Driven Counterfactual Testing

arXiv 2609.17953 ↗First posted 16 Sep 2026cs.CV cs.AIcited by 0

On the map: Multimodal LLMs / vision-language models → VLM benchmarks and visual reasoning

Abstract (arXiv)

Vision-Language Models (VLMs) can produce Natural Language Explanations (NLEs) that sound plausible yet remain inconsistent with the visual evidence they cite. We present Explanation-Driven Counterfactual Testing (EDCT), an intervention-based protocol that extracts visual concepts cited in a model's explanation, applies verified minimal edits to them, and tests whether the resulting answer and explanation remain consistent with the edited image. Using this protocol, we create EDCT-Bench, a comprehensive benchmark spanning three complementary domains: knowledge-intensive visual question answering (OK-VQA), safety-critical driving (DriveLM), and 3D spatial reasoning (3DSRBench). Across the evaluated VLMs, EDCT reveals substantial faithfulness gaps, with models frequently producing responses inconsistent with verified visual changes. Finally, our fine-tuning study suggests that EDCT-generated counterfactuals provide high-impact training signals.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in VLM benchmarks and visual reasoning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 35 of its 53 references that are in the map. First posted on arXiv 16 Sep 2026.

Builds on

35 of 53 references.

Closest ancestors · 6

Same direction, newest first

  1. 2026Are VLMs Seeing or Just Saying? Uncovering the Illusion of Visual Re-examination15 May
  2. 2025Journey Before Destination: On the importance of Visual Faithfulness in Slow Thinking13 Dec
  3. 2025V-SEAM: Visual Semantic Editing and Attention Modulating for Causal Interpretability of Vision-Language Models18 Sep
  4. 2025CF-VLM:CounterFactual Vision-Language Fine-tuning10 Jun
  5. 2024Performance Gap in Entity Knowledge Extraction Across Modalities in Vision Language Models18 Dec
  6. 2023MathVista: Evaluating Mathematical Reasoning of Foundation Models in Visual Contexts3 Oct

Other references · 29

  1. 2025Qwen3 Technical Reportcited by 8,820
  2. 2020Measuring Massive Multitask Language Understandingcited by 7,963
  3. 2016Grad-CAM: Visual Explanations from Deep Networks via Gradient-based Localizationcited by 6,733
  4. 2025Gemini 2.5: Pushing the Frontier with Advanced Reasoning, Multimodality, Long Context, and Next Generation Agentic Capabilitiescited by 3,796
  5. 2023QLoRA: Efficient Finetuning of Quantized LLMscited by 3,418
  6. 2017Axiomatic Attribution for Deep Networkscited by 3,384
  7. 2024SAM 2: Segment Anything in Images and Videoscited by 2,863
  8. 2023MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGIcited by 2,080
  9. 2025Gemma 3 Technical Reportcited by 1,824
  10. 2024A Survey on LLM-as-a-Judgecited by 1,423
  11. 2018Sanity Checks for Saliency Mapscited by 1,156
  12. 2019OK-VQA: A Visual Question Answering Benchmark Requiring External Knowledgecited by 1,121
17 more
  1. 2020Towards Faithfully Interpretable NLP Systems: How should we define and evaluate faithfulness?cited by 599
  2. 2023DriveLM: Driving with Graph Visual Question Answeringcited by 536
  3. 2024LLMs-as-Judges: A Comprehensive Survey on LLM-based Evaluation Methodscited by 465
  4. 2021Generic Attention-model Explainability for Interpreting Bi-Modal and Encoder-Decoder Transformerscited by 320
  5. 2019Counterfactual Visual Explanationscited by 313
  6. 2024Pixtral 12Bcited by 179
  7. 2025See What You Are Told: Visual Attention Sink in Large Multimodal Modelscited by 132
  8. 2021Counterfactual Explanations Can Be Manipulatedcited by 113
  9. 2023Faithfulness Tests for Natural Language Explanationscited by 107
  10. 2024Faithfulness vs. Plausibility: On the (Un)Reliability of Explanations from Large Language Modelscited by 105
  11. 2024LLM-Rubric: A Multidimensional, Calibrated Approach to Automated Evaluation of Natural Language Textscited by 87
  12. 2018Faithful Multimodal Explanation for Visual Question Answeringcited by 64
  13. 2024CounterCurate: Enhancing Physical and Semantic Visio-Linguistic Compositional Reasoning via Counterfactual Examplescited by 35
  14. 2025Measuring the Faithfulness of Thinking Drafts in Large Reasoning Modelscited by 23
  15. 2024VALOR-EVAL: Holistic Coverage and Faithfulness Evaluation of Large Vision-Language Modelscited by 16
  16. 2025A Closer Look at Bias and Chain-of-Thought Faithfulness of Large (Vision) Language Modelscited by 12
  17. 2025Counterfactual Evaluation for Blind Attack Detection in LLM-based Evaluation Systemscited by 3

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025Explanation-Driven Counterfactual Testing for Faithfulness in Vision-Language Model Explanations
  2. 2026Evidence-RL: Towards Evidence-intensive Visual Reasoning
  3. 2026Counterfactual Tests for Measuring Chain-of-Thought Faithfulness in Visual Language Models
  4. 2026Seeing What Is Actually There: PriVE-Bench and PriVE-Tools for Counterfactual Evaluation of Agentic Visual Evidence in VLMs
  5. 2025Distilling Counterfactual Reasoning from Language to Vision: Causal Graph Guided Post-Training for Video Understanding
  6. 2025V-CECE: Visual Counterfactual Explanations via Conceptual Edits
  7. 2025Believing without Seeing: Quality Scores for Contextualizing Vision-Language Model Explanations
  8. 2026Look Where It Counts: A Free, Label-Free Visual Evidence Signal for Fine-Grained Vision-Language Reasoning