mlsanityData to 7 Oct 2026

Causal Attention for Vision-Language Tasks

arXiv 2103.03493 ↗First posted 5 Mar 2021cs.CVcited by 111

On the map: Multimodal LLMs / vision-language models → VLM benchmarks and visual reasoning

Abstract (arXiv)

We present a novel attention mechanism: Causal Attention (CATT), to remove the ever-elusive confounding effect in existing attention-based vision-language models. This effect causes harmful bias that misleads the attention module to focus on the spurious correlations in training data, damaging the model generalization. As the confounder is unobserved in general, we use the front-door adjustment to realize the causal intervention, which does not require any knowledge on the confounder. Specifically, CATT is implemented as a combination of 1) In-Sample Attention (IS-ATT) and 2) Cross-Sample Attention (CS-ATT), where the latter forcibly brings other samples into every IS-ATT, mimicking the causal intervention. CATT abides by the Q-K-V convention and hence can replace any attention module such as top-down attention and self-attention in Transformers. CATT improves various popular attention-based vision-language models by considerable margins. In particular, we show that CATT has great potential in large-scale pre-training, e.g., it can promote the lighter LXMERT~citetan2019lxmert, which uses fewer data and less computational power, comparable to the heavier UNITER~citechen2020uniter. Code is published in urlhttps://github.com/yangxuntu/catt.

Key facts. Cited by 111 papers in the map (data through 7 Oct 2026). Placed in VLM benchmarks and visual reasoning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 61 of its 86 references that are in the map. First posted on arXiv 5 Mar 2021.

Builds on

61 of 86 references.

References · 61

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  4. 2015Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networkscited by 9,617
  5. 2020End-to-End Object Detection with Transformerscited by 7,384
  6. 2014Neural Machine Translation by Jointly Learning to Align and Translatecited by 4,530
  7. 2016Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotationscited by 2,646
  8. 2016Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answeringcited by 2,498
  9. 2015VQA: Visual Question Answeringcited by 2,489
  10. 2017Non-local Neural Networkscited by 2,290
  11. 2019ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Taskscited by 2,219
  12. 2014CIDEr: Consensus-based Image Description Evaluationcited by 2,027
38 more
  1. 2019LXMERT: Learning Cross-Modality Encoder Representations from Transformerscited by 1,548
  2. 2015Microsoft COCO Captions: Data Collection and Evaluation Servercited by 1,447
  3. 2015Show, Attend and Tell: Neural Image Caption Generation with Visual Attentioncited by 1,421
  4. 2018Relational inductive biases, deep learning, and graph networkscited by 1,341
  5. 2017Bottom-Up and Top-Down Attention for Image Captioning and Visual Question Answeringcited by 1,310
  6. 2016Man is to Computer Programmer as Woman is to Homemaker? Debiasing Word Embeddingscited by 1,304
  7. 2019VisualBERT: A Simple and Performant Baseline for Vision and Languagecited by 1,291
  8. 2020Oscar: Object-Semantics Aligned Pre-training for Vision-Language Taskscited by 1,213
  9. 2014Deep Visual-Semantic Alignments for Generating Image Descriptionscited by 966
  10. 2020StereoSet: Measuring stereotypical bias in pretrained language modelscited by 911
  11. 2014Show and Tell: A Neural Image Caption Generatorcited by 877
  12. 2016SPICE: Semantic Propositional Image Caption Evaluationcited by 820
  13. 2018Object Hallucination in Image Captioningcited by 661
  14. 2016Self-critical Sequence Training for Image Captioningcited by 544
  15. 2019Unified Vision-Language Pre-Training for Image Captioning and VQAcited by 494
  16. 2018A Corpus for Reasoning About Natural Language Grounded in Photographscited by 457
  17. 2020Unbiased Scene Graph Generation from Biased Trainingcited by 355
  18. 2019Meshed-Memory Transformer for Image Captioningcited by 343
  19. 2019Deep Modular Co-Attention Networks for Visual Question Answeringcited by 338
  20. 2019Attention on Attention for Image Captioningcited by 289
  21. 2016Hierarchical Question-Image Co-Attention for Visual Question Answeringcited by 278
  22. 201912-in-1: Multi-Task Vision and Language Representation Learningcited by 278
  23. 2019Measuring Bias in Contextualized Word Representationscited by 275
  24. 2015Stacked Attention Networks for Image Question Answeringcited by 273
  25. 2020Counterfactual VQA: A Cause-Effect Look at Language Biascited by 266
  26. 2016Image Captioning with Semantic Attentioncited by 260
  27. 2018Auto-Encoding Scene Graphs for Image Captioningcited by 254
  28. 2018Women also Snowboard: Overcoming Bias in Captioning Modelscited by 236
  29. 2020Causal Intervention for Weakly-Supervised Semantic Segmentationcited by 229
  30. 2018Exploring Visual Relationship for Image Captioningcited by 215
  31. 2020ERNIE-ViL: Knowledge Enhanced Vision-Language Representations Through Scene Graphcited by 211
  32. 2019A Meta-Transfer Objective for Learning to Disentangle Causal Mechanismscited by 158
  33. 2017CausalGAN: Learning Causal Implicit Generative Models with Adversarial Trainingcited by 152
  34. 2017MUTAN: Multimodal Tucker Fusion for Visual Question Answeringcited by 137
  35. 2021Distilling Causal Effect of Data in Class-Incremental Learningcited by 117
  36. 2018Dynamic Fusion with Intra- and Inter- Modality Attention Flow for Visual Question Answeringcited by 113
  37. 2020Interventional Few-Shot Learningcited by 112
  38. 2018Robustly Disentangled Causal Mechanisms: Validating Deep Representations for Interventional Robustnesscited by 108

50 most cited of 61 All 61 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 10; 2022: 20; 2023: 20; 2024: 20; 2025: 24; 2026: 17.

111 papers.

Sort:

By area:

  1. 2022Multimodal Learning with Transformers: A Surveycited by 399
  2. 2021TransVG: End-to-End Visual Grounding with Transformerscited by 269
  3. 2021Causal Attention for Interpretable and Generalizable Graph Classificationcited by 108
  4. 2022VindLU: A Recipe for Effective Video-and-Language Pretrainingcited by 87
  5. 2021Causal Attention for Unbiased Visual Recognitioncited by 80
  6. 2022Cross-Modal Causal Relational Reasoning for Event-Level Visual Question Answeringcited by 57
  7. 2024Vision-and-Language Navigation via Causal Learningcited by 51
  8. 2023Transformers in Speech Processing: A Surveycited by 38
  9. 2024Causal Prompting: Debiasing Large Language Model Prompting based on Front-Door Adjustmentcited by 29
  10. 2021Introspective Distillation for Robust Question Answeringcited by 25
  11. 2022Causal Reasoning Meets Visual Representation Learning: A Prospective Studycited by 24
  12. 2024Causality for Large Language Modelscited by 24
38 more
  1. 2022Neural Integral Equationscited by 23
  2. 2024MECD: Unlocking Multi-Event Causal Discovery in Video Reasoningcited by 23
  3. 2022Equivariant and Invariant Grounding for Video Question Answeringcited by 22
  4. 2022Debiasing Methods for Fairer Neural Models in Vision and Language Research: A Surveycited by 22
  5. 2023Debiased Fine-Tuning for Vision-language Models by Prompt Regularizationcited by 20
  6. 2021Cross-Domain Empirical Risk Minimization for Unbiased Long-tailed Classificationcited by 19
  7. 2021Data Efficient Masked Language Modeling for Vision and Languagecited by 18
  8. 2023Causal Unsupervised Semantic Segmentationcited by 18
  9. 2022DALL-E for Detection: Language-driven Compositional Image Synthesis for Object Detectioncited by 17
  10. 2022Cross-modal Attention Congruence Regularization for Vision-Language Relation Alignmentcited by 17
  11. 2023Learning Distortion Invariant Representation for Image Restoration from A Causality Perspectivecited by 17
  12. 2023Visual Causal Scene Refinement for Video Question Answeringcited by 16
  13. 2023Transforming Visual Scene Graphs to Image Captionscited by 15
  14. 2021Causal Incremental Graph Convolution for Recommender System Retrainingcited by 14
  15. 2023Cross-Modal Causal Intervention for Medical Report Generationcited by 14
  16. 2024Causal Mode Multiplexer: A Novel Framework for Unbiased Multispectral Pedestrian Detectioncited by 13
  17. 2024Causality-inspired Discriminative Feature Learning in Triple Domains for Gait Recognitioncited by 13
  18. 2021Semi-Autoregressive Transformer for Image Captioningcited by 12
  19. 2024Enhancing Vision-Language Pre-training with Rich Supervisionscited by 12
  20. 2024Visual Language Model based Cross-modal Semantic Communication Systemscited by 12
  21. 2025Do You Keep an Eye on What I Ask? Mitigating Multimodal Hallucination via Attention-Guided Ensemble Decodingcited by 12
  22. 2023Simple Token-Level Confidence Improves Caption Correctnesscited by 11
  23. 2024Causality-Aware Spatiotemporal Graph Neural Networks for Spatiotemporal Time Series Imputationcited by 11
  24. 2024De-confounded Data-free Knowledge Distillation for Handling Distribution Shiftscited by 11
  25. 2021Confounder Identification-free Causal Visual Feature Learningcited by 10
  26. 2022Clinically-Inspired Multi-Agent Transformers for Disease Trajectory Forecasting from Multimodal Datacited by 10
  27. 2023Beyond Generation: Harnessing Text to Image Models for Object Detection and Segmentationcited by 10
  28. 2024Debiased Multimodal Understanding for Human Language Sequencescited by 10
  29. 2025CleanPose: Category-Level Object Pose Estimation via Causal Learning and Knowledge Distillationcited by 9
  30. 2022Domain Generalization via Contrastive Causal Learningcited by 8
  31. 2022Causal Inference via Style Transfer for Out-of-distribution Generalisationcited by 8
  32. 2024Time-Causal VAE: Robust Financial Time Series Generatorcited by 8
  33. 2025Resilient Contrastive Pre-training under Non-Stationary Driftcited by 7
  34. 2025CAUSAL3D: A Comprehensive Benchmark for Causal Learning from Visual Datacited by 7
  35. 2022Counterfactual Multihop QA: A Cause-Effect Approach for Reducing Disconnected Reasoningcited by 6
  36. 2022Learning by Hallucinating: Vision-Language Pre-training with Weak Supervisioncited by 6
  37. 2023A Survey of Methods, Challenges and Perspectives in Causalitycited by 6
  38. 2024Towards Deconfounded Image-Text Matching with Causal Inferencecited by 6

50 most cited of 111 All 111 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025Causal Tracing of Object Representations in Large Vision Language Models: Mechanistic Interpretability and Hallucination Mitigation
  2. 2024Looking Beyond Text: Reducing Language bias in Large Vision-Language Models via Multimodal Dual-Attention and Soft-Image Guidance
  3. 2026Disentangling Semantic Attention from Structural Bias in the Attention Manifold
  4. 2025What's Missing in Vision-Language Models? Probing Their Struggles with Causal Order Reasoning
  5. 2025V-SEAM: Visual Semantic Editing and Attention Modulating for Causal Interpretability of Vision-Language Models
  6. 2025Seeing is Understanding: Unlocking Causal Attention into Modality-Mutual Attention for Multimodal LLMs
  7. 2025CausalVLBench: Benchmarking Visual Causal Reasoning in Large Vision-Language Models
  8. 2026Visual Attention Faithfulness in Vision-Language Models is Heterogeneous