mlsanityData to 14 Oct 2026

Beyond Spurious Signals: Debiasing Multimodal Large Language Models via Counterfactual Inference and Adaptive Expert Routing

arXiv 2509.15361 ↗First posted 18 Sep 2025cs.CL cs.AI cs.MMcited by 5

On the map: LLM reasoning → Multimodal reasoning

also in 2 more
Abstract (arXiv)

Multimodal Large Language Models (MLLMs) have shown substantial capabilities in integrating visual and textual information, yet frequently rely on spurious correlations, undermining their robustness and generalization in complex multimodal reasoning tasks. This paper addresses the critical challenge of superficial correlation bias in MLLMs through a novel causal mediation-based debiasing framework. Specially, we distinguishing core semantics from spurious textual and visual contexts via counterfactual examples to activate training-stage debiasing and employ a Mixture-of-Experts (MoE) architecture with dynamic routing to selectively engages modality-specific debiasing experts. Empirical evaluation on multimodal sarcasm detection and sentiment analysis tasks demonstrates that our framework significantly surpasses unimodal debiasing strategies and existing state-of-the-art models.

Key facts. Cited by 5 papers in the map (data through 14 Oct 2026). Placed in Multimodal reasoning, one of 17 directions in LLM reasoning. Builds on 35 of its 63 references that are in the map. First posted on arXiv 18 Sep 2025.

Builds on

35 of 63 references.

Closest ancestors · 3

Same direction, newest first

  1. 2024Expanding Performance Boundaries of Open-Source Multimodal Models with Model, Data, and Test-Time Scaling6 Dec
  2. 2024Enhancing Semantics in Multimodal Chain of Thought via Soft Negative Sampling16 May
  3. 2022Winoground: Probing Vision and Language Models for Visio-Linguistic Compositionality7 Apr

Other references · 32

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  2. 2024Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolutioncited by 4,234
  3. 2023InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Taskscited by 2,565
  4. 2024LlamaFactory: Unified Efficient Fine-Tuning of 100+ Language Modelscited by 1,464
  5. 2019ZeRO: Memory Optimizations Toward Training Trillion Parameter Modelscited by 1,301
  6. 2023Mitigating Object Hallucinations in Large Vision-Language Models through Visual Contrastive Decodingcited by 688
  7. 2024Hallucination of Multimodal Large Language Models: A Surveycited by 388
  8. 2017Don't Just Assume; Look and Answer: Overcoming Priors for Visual Question Answeringcited by 339
  9. 2020Counterfactual VQA: A Cause-Effect Look at Language Biascited by 266
  10. 2021Causal Inference in Natural Language Processing: Estimation, Prediction, Interpretation and Beyondcited by 178
  11. 2019Towards Multimodal Sarcasm Detection (An _Obviously_ Perfect Paper)cited by 174
  12. 2020Counterfactual Samples Synthesizing for Robust Visual Question Answeringcited by 153
20 more
  1. 2019Towards Causal VQA: Revealing and Reducing Spurious Correlations by Invariant and Covariant Semantic Editingcited by 108
  2. 2017FOIL it! Find One mismatch between Image and Language captioncited by 95
  3. 2023Towards Vision-Language Mechanistic Interpretability: A Causal Tracing Tool for BLIPcited by 62
  4. 2024A Comprehensive Review of Multimodal Large Language Models: Performance and Challenges Across Different Taskscited by 57
  5. 2023SocialCounterfactuals: Probing and Mitigating Intersectional Social Biases in Vision-Language Models with Counterfactual Examplescited by 53
  6. 2023COCO-Counterfactuals: Automatically Constructed Counterfactual Examples for Image-Text Pairscited by 39
  7. 2024MM-SpuBench: Towards Better Understanding of Spurious Biases in Multimodal LLMscited by 33
  8. 2024Looking Beyond Text: Reducing Language bias in Large Vision-Language Models via Multimodal Dual-Attention and Soft-Image Guidancecited by 32
  9. 2023MMSD2.0: Towards a Reliable Multi-modal Sarcasm Detection Systemcited by 31
  10. 2022Towards Multi-Modal Sarcasm Detection via Hierarchical Congruity Modeling with Knowledge Enhancementcited by 27
  11. 2024Towards Multimodal Sentiment Analysis Debiasing via Bias Purificationcited by 27
  12. 2024Robust Emotion Recognition in Context Debiasingcited by 24
  13. 2022CLMLF:A Contrastive Learning and Multi-Layer Fusion Method for Multimodal Sentiment Detectioncited by 21
  14. 2024A Survey of Multimodal Sarcasm Detectioncited by 19
  15. 2024Images Speak Louder than Words: Understanding and Mitigating Bias in Vision-Language Model from a Causal Mediation Perspectivecited by 13
  16. 2025Aligning Multimodal LLM with Human Preference: A Surveycited by 13
  17. 2025SpurLens: Automatic Detection of Spurious Cues in Multimodal LLMscited by 7
  18. 2023Debiasing Multimodal Models via Causal Information Minimizationcited by 4
  19. 2021Counterfactual Adversarial Learning with Representation Interpolationcited by 2
  20. 2024Multimodal Sentiment Analysis Based on Causal Reasoningcited by 1

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 1; 2026: 4.

5 papers.

  1. 2025OmniMER: Auxiliary-Enhanced LLM Adaptation for Indonesian Multimodal Emotion Recognitioncited by 0
  2. 2026QASA: Quality-Aware Semantic Augmentation for Robust Multimodal Sentiment Analysiscited by 0
  3. 2026UGID: Unified Graph Isomorphism for Debiasing Large Language Modelscited by 0
  4. 2026Sensing Which Modality Matters: Evidence-Gated Regularization for Robust VLA Policiescited by 0
  5. 2026Beyond Text: LLM-Based Dimensional Emotion Evaluation in Multimodal Dialoguecited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Can Large Vision-Language Models Understand Multimodal Sarcasm?
  2. 2025Think Twice Before You Judge: Mixture of Dual Reasoning Experts for Multimodal Sarcasm Detection
  3. 2026MuVaC: A Variational Causal Framework for Multimodal Sarcasm Understanding in Dialogues
  4. 2026URMF: Uncertainty-aware Robust Multimodal Fusion for Multimodal Sarcasm Detection
  5. 2024Debiasing Multimodal Large Language Models via Penalization of Language Priors
  6. 2024CofiPara: A Coarse-to-fine Paradigm for Multimodal Sarcasm Target Identification with Large Multimodal Models
  7. 2025Evaluating Multimodal Large Language Models on Spoken Sarcasm Understanding
  8. 2025Evaluating Open-Source Vision-Language Models for Multimodal Sarcasm Detection