# Cross-modal Attention Congruence Regularization for Vision-Language Relation Alignment

arXiv: https://arxiv.org/abs/2212.10549 · HTML: https://mlsanity.com/paper/2212.10549/

## Key facts

- Cited by 17 papers in the map (data through 14 Oct 2026).
- Placed in MLLM training and visual instruction tuning, one of 16 directions in Multimodal LLMs / vision-language models.
- Builds on 23 of its 41 references that are in the map.
- First posted on arXiv 20 Dec 2022.

## Placed in

- Multimodal LLMs / vision-language models / [MLLM training and visual instruction tuning](https://mlsanity.com/mllm/mllm-training-and-visual-instruction-tuning/)

## Builds on (most cited first, 23 of 23)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [LXMERT: Learning Cross-Modality Encoder Representations from Transformers](https://mlsanity.com/paper/1908.07490/) (2019)
- [Quantifying Attention Flow in Transformers](https://mlsanity.com/paper/2005.00928/) (2020)
- [Winoground: Probing Vision and Language Models for Visio-Linguistic Compositionality](https://mlsanity.com/paper/2204.03162/) (2022)
- [Neural Module Networks](https://mlsanity.com/paper/1511.02799/) (2015)
- [ERNIE-ViL: Knowledge Enhanced Vision-Language Representations Through Scene Graph](https://mlsanity.com/paper/2006.16934/) (2020)
- [Causal Attention for Vision-Language Tasks](https://mlsanity.com/paper/2103.03493/) (2021)
- [Multimodal Pretraining Unmasked: A Meta-Analysis and a Unified Framework of Vision-and-Language BERTs](https://mlsanity.com/paper/2011.15124/) (2020)
- [Probing Inter-modality: Visual Parsing with Self-Attention for Vision-Language Pre-training](https://mlsanity.com/paper/2106.13488/) (2021)
- [VL-InterpreT: An Interactive Visualization Tool for Interpreting Vision-Language Transformers](https://mlsanity.com/paper/2203.17247/) (2022)
- [Syntax-BERT: Improving Pre-trained Transformers with Syntax Trees](https://mlsanity.com/paper/2103.04350/) (2021)
- [ROSITA: Enhancing Vision-and-Language Semantic Alignments via Cross- and Intra-modal Knowledge Integration](https://mlsanity.com/paper/2108.07073/) (2021)
- [Improving BERT with Syntax-aware Local Attention](https://mlsanity.com/paper/2012.15150/) (2020)
- [Aligning Linguistic Words and Visual Semantic Units for Image Captioning](https://mlsanity.com/paper/1908.02127/) (2019)
- [From Balustrades to Pierre Vinken: Looking for Syntax in Transformer Self-Attentions](https://mlsanity.com/paper/1906.01958/) (2019)
- [VLGrammar: Grounded Grammar Induction of Vision and Language](https://mlsanity.com/paper/2103.12975/) (2021)
- [Learning Relation Alignment for Calibrated Cross-modal Retrieval](https://mlsanity.com/paper/2105.13868/) (2021)
- [Auto-Parsing Network for Image Captioning and Visual Question Answering](https://mlsanity.com/paper/2108.10568/) (2021)
- [SGEITL: Scene Graph Enhanced Image-Text Learning for Visual Commonsense Reasoning](https://mlsanity.com/paper/2112.08587/) (2021)
- [Single-Stream Multi-Level Alignment for Vision-Language Pretraining](https://mlsanity.com/paper/2203.14395/) (2022)
- [Finding Structural Knowledge in Multimodal-BERT](https://mlsanity.com/paper/2203.09306/) (2022)
- [Cross-Modal Alignment Learning of Vision-Language Conceptual Systems](https://mlsanity.com/paper/2208.01744/) (2022)

## Built on by (most cited first, 17 of 17)

- [What You See is What You Read? Improving Text-Image Alignment Evaluation](https://mlsanity.com/paper/2305.10400/) (2023)
- [VisIT-Bench: A Benchmark for Vision-Language Instruction Following Inspired by Real-World Use](https://mlsanity.com/paper/2308.06595/) (2023)
- [EventGPT: Event Stream Understanding with Multimodal Large Language Models](https://mlsanity.com/paper/2412.00832/) (2024)
- [Vision-Language Dataset Distillation](https://mlsanity.com/paper/2308.07545/) (2023)
- [Progressive Compositionality in Text-to-Image Generative Models](https://mlsanity.com/paper/2410.16719/) (2024)
- [Simple Token-Level Confidence Improves Caption Correctness](https://mlsanity.com/paper/2305.07021/) (2023)
- [Encoding and Controlling Global Semantics for Long-form Video Question Answering](https://mlsanity.com/paper/2405.19723/) (2024)
- [Prompting Large Vision-Language Models for Compositional Reasoning](https://mlsanity.com/paper/2401.11337/) (2024)
- [SEAM: Semantically Equivalent Across Modalities Benchmark for Vision-Language Models](https://mlsanity.com/paper/2508.18179/) (2025)
- [Evaluating Compositional Generalisation in VLMs and Diffusion Models](https://mlsanity.com/paper/2508.20783/) (2025)
- [Attention Misses Visual Risk: Risk-Adaptive Steering for Multimodal Safety Alignment](https://mlsanity.com/paper/2510.13698/) (2025)
- [OLIVE: Object Level In-Context Visual Embeddings](https://mlsanity.com/paper/2406.00872/) (2024)
- [Watch, Listen, Understand, Mislead: Tri-modal Adversarial Attacks on Short Videos for Content Appropriateness Evaluation](https://mlsanity.com/paper/2507.11968/) (2025)
- [Multi-Modal Continual Learning via Cross-Modality Adapters and Representation Alignment with Knowledge Preservation](https://mlsanity.com/paper/2511.06723/) (2025)
- [Targeted Image Data Augmentation Increases Basic Skills Captioning Robustness](https://mlsanity.com/paper/2309.15991/) (2023)
- [Multi-Domain EEG Representation Learning with Orthogonal Mapping and Attention-based Fusion for Cognitive Load Classification](https://mlsanity.com/paper/2511.12394/) (2025)
- [Token-level Response-visual Attention Guidance for Multimodal LLMs Knowledge Distillation](https://mlsanity.com/paper/2607.02593/) (2026)

## Abstract (arXiv)

Despite recent progress towards scaling up multimodal vision-language models, these models are still known to struggle on compositional generalization benchmarks such as Winoground. We find that a critical component lacking from current vision-language models is relation-level alignment: the ability to match directional semantic relations in text (e.g., "mug in grass") with spatial relationships in the image (e.g., the position of the mug relative to the grass). To tackle this problem, we show that relation alignment can be enforced by encouraging the directed language attention from 'mug' to 'grass' (capturing the semantic relation 'in') to match the directed visual attention from the mug to the grass. Tokens and their corresponding objects are softly identified using the cross-modal attention. We prove that this notion of soft relation alignment is equivalent to enforcing congruence between vision and language attention matrices under a 'change of basis' provided by the cross-modal attention matrix. Intuitively, our approach projects visual attention into the language attention space to calculate its divergence from the actual language attention, and vice versa. We apply our Cross-modal Attention Congruence Regularization (CACR) loss to UNITER and improve on the state-of-the-art approach to Winoground.