mlsanityData to 7 Oct 2026

Debiased Multimodal Understanding for Human Language Sequences

arXiv 2403.05025 ↗First posted 8 Mar 2024cs.AIcited by 10

Not on the map. Why?

Abstract (arXiv)

Human multimodal language understanding (MLU) is an indispensable component of expression analysis (e.g., sentiment or humor) from heterogeneous modalities, including visual postures, linguistic contents, and acoustic behaviours. Existing works invariably focus on designing sophisticated structures or fusion strategies to achieve impressive improvements. Unfortunately, they all suffer from the subject variation problem due to data distribution discrepancies among subjects. Concretely, MLU models are easily misled by distinct subjects with different expression customs and characteristics in the training data to learn subject-specific spurious correlations, limiting performance and generalizability across new subjects. Motivated by this observation, we introduce a recapitulative causal graph to formulate the MLU procedure and analyze the confounding effect of subjects. Then, we propose SuCI, a simple yet effective causal intervention module to disentangle the impact of subjects acting as unobserved confounders and achieve model training via true causal effects. As a plug-and-play component, SuCI can be widely applied to most methods that seek unbiased predictions. Comprehensive experiments on several MLU benchmarks clearly show the effectiveness of the proposed module.

Key facts. Cited by 10 papers in the map (data through 7 Oct 2026). Builds on 25 of its 48 references that are in the map. First posted on arXiv 8 Mar 2024.

Builds on

25 of 48 references.

References · 25

  1. 2016Gaussian Error Linear Units (GELUs)cited by 3,431
  2. 2015Show, Attend and Tell: Neural Image Caption Generation with Visual Attentioncited by 1,421
  3. 2019Multimodal Transformer for Unaligned Multimodal Language Sequencescited by 637
  4. 2017Tensor Fusion Network for Multimodal Sentiment Analysiscited by 429
  5. 2020Unbiased Scene Graph Generation from Biased Trainingcited by 355
  6. 2018Deep Facial Expression Recognition: A Surveycited by 269
  7. 2018Efficient Low-rank Multimodal Fusion with Modality-Specific Factorscited by 268
  8. 2020MISA: Modality-Invariant and -Specific Representations for Multimodal Sentiment Analysiscited by 262
  9. 2016MOSI: Multimodal Corpus of Sentiment Intensity and Subjectivity Analysis in Online Opinion Videoscited by 193
  10. 2021Learning Modality-Specific Representations with Self-Supervised Multi-Task Learning for Multimodal Sentiment Analysiscited by 179
  11. 2018Learning Factorized Multimodal Representationscited by 171
  12. 2018Found in Translation: Learning Robust Joint Representations by Cyclic Translations Between Modalitiescited by 149
13 more
  1. 2021MultiBench: Multiscale Benchmarks for Multimodal Representation Learningcited by 143
  2. 2019UR-FUNNY: A Multimodal Language Dataset for Understanding Humorcited by 130
  3. 2021Improving Multimodal Fusion with Hierarchical Mutual Information Maximization for Multimodal Sentiment Analysiscited by 130
  4. 2021Causal Attention for Vision-Language Taskscited by 111
  5. 2018Words Can Shift: Dynamically Adjusting Word Representations Using Nonverbal Behaviorscited by 89
  6. 2020Deconfounded Image Captioning: A Causal Retrospectcited by 65
  7. 2023Generalized Video Anomaly Event Detection: Systematic Taxonomy and Comparison of Deep Modelscited by 59
  8. 2019Causal Reasoning from Meta-reinforcement Learningcited by 53
  9. 2023Spatio-Temporal Domain Awareness for Multi-Agent Collaborative Perceptioncited by 52
  10. 2022CubeMLP: An MLP-based Model for Multimodal Sentiment Analysis and Depression Estimationcited by 39
  11. 2020Counterfactually-Augmented SNLI Training Data Does Not Yield Better Generalization Than Unaugmented Datacited by 27
  12. 2021De-biasing Distantly Supervised Named Entity Recognition via Causal Interventioncited by 24
  13. 2022Causal Intervention for Subject-Deconfounded Facial Action Unit Recognitioncited by 11

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 3; 2025: 4; 2026: 3.

10 papers.

  1. 2024Correlation-Decoupled Knowledge Distillation for Multimodal Sentiment Analysis with Incomplete Modalitiescited by 32
  2. 2024De-confounded Data-free Knowledge Distillation for Handling Distribution Shiftscited by 11
  3. 2024Towards Context-Aware Emotion Recognition Debiasing from a Causal Demystification Perspective via De-confounded Trainingcited by 11
  4. 2025SAIL-Embedding Technical Report: Omni-modal Embedding Foundation Modelcited by 10
  5. 2025Disentangling Bias by Modeling Intra- and Inter-modal Causal Attention for Multimodal Sentiment Analysiscited by 2
  6. 2025Towards Minimal Causal Representations for Human Multimodal Language Understandingcited by 2
  7. 2026Multi-Modal Image Fusion via Intervention-Stable Feature Learningcited by 1
  8. 2025Drift-Aware Federated Learning: A Causal Perspectivecited by 0
  9. 2026Learning Invariant Modality Representation for Robust Multimodal Learning from a Causal Inference Perspectivecited by 0
  10. 2026Reliability-aware Cross-sample Enhancement for Robust Multimodal Sentiment Analysiscited by 0