# Understanding the Emergence of Multimodal Representation Alignment

arXiv: https://arxiv.org/abs/2502.16282 · HTML: https://mlsanity.com/paper/2502.16282/

## Key facts

- Cited by 31 papers in the map (data through 14 Oct 2026).
- Builds on 40 of its 61 references that are in the map.
- First posted on arXiv 22 Feb 2025.

## Builds on (most cited first, 30 of 40)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://mlsanity.com/paper/2307.09288/) (2023)
- [ImageNet Large Scale Visual Recognition Challenge](https://mlsanity.com/paper/1409.0575/) (2014)
- [DINOv2: Learning Robust Visual Features without Supervision](https://mlsanity.com/paper/2304.07193/) (2023)
- [Masked Autoencoders Are Scalable Vision Learners](https://mlsanity.com/paper/2111.06377/) (2021)
- [Flamingo: a Visual Language Model for Few-Shot Learning](https://mlsanity.com/paper/2204.14198/) (2022)
- [Zero-Shot Text-to-Image Generation](https://mlsanity.com/paper/2102.12092/) (2021)
- [Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervision](https://mlsanity.com/paper/2102.05918/) (2021)
- [BLOOM: A 176B-Parameter Open-Access Multilingual Language Model](https://mlsanity.com/paper/2211.05100/) (2022)
- [Similarity of Neural Network Representations Revisited](https://mlsanity.com/paper/1905.00414/) (2019)
- [Make-A-Video: Text-to-Video Generation without Text-Video Data](https://mlsanity.com/paper/2209.14792/) (2022)
- [Align before Fuse: Vision and Language Representation Learning with Momentum Distillation](https://mlsanity.com/paper/2107.07651/) (2021)
- [Multimodal Machine Learning: A Survey and Taxonomy](https://mlsanity.com/paper/1705.09406/) (2017)
- [What Makes for Good Views for Contrastive Learning?](https://mlsanity.com/paper/2005.10243/) (2020)
- [VideoBERT: A Joint Model for Video and Language Representation Learning](https://mlsanity.com/paper/1904.01766/) (2019)
- [Fully-Convolutional Siamese Networks for Object Tracking](https://mlsanity.com/paper/1606.09549/) (2016)
- [The Platonic Representation Hypothesis](https://mlsanity.com/paper/2405.07987/) (2024)
- [Insights on representational similarity in neural networks with canonical correlation](https://mlsanity.com/paper/1806.05759/) (2018)
- [WIT: Wikipedia-based Image Text Dataset for Multimodal Multilingual Machine Learning](https://mlsanity.com/paper/2103.01913/) (2021)
- [Gated Multimodal Units for Information Fusion](https://mlsanity.com/paper/1702.01992/) (2017)
- [Understanding image representations by measuring their equivariance and equivalence](https://mlsanity.com/paper/1411.5908/) (2014)
- [Convergent Learning: Do different neural networks learn the same representations?](https://mlsanity.com/paper/1511.07543/) (2015)
- [Revisiting Model Stitching to Compare Neural Representations](https://mlsanity.com/paper/2106.07682/) (2021)
- [MOSI: Multimodal Corpus of Sentiment Intensity and Subjectivity Analysis in Online Opinion Videos](https://mlsanity.com/paper/1606.06259/) (2016)
- [Towards Multimodal Sarcasm Detection (An _Obviously_ Perfect Paper)](https://mlsanity.com/paper/1906.01815/) (2019)
- [On Deep Multi-View Representation Learning: Objectives and Optimization](https://mlsanity.com/paper/1602.01024/) (2016)
- [Making Sense of Vision and Touch: Self-Supervised Learning of Multimodal Representations for Contact-Rich Tasks](https://mlsanity.com/paper/1810.10191/) (2018)
- [Contrastive learning, multi-view redundancy, and linear models](https://mlsanity.com/paper/2008.10150/) (2020)
- [MultiBench: Multiscale Benchmarks for Multimodal Representation Learning](https://mlsanity.com/paper/2107.07502/) (2021)
- [UR-FUNNY: A Multimodal Language Dataset for Understanding Humor](https://mlsanity.com/paper/1904.06618/) (2019)

## Built on by (most cited first, 30 of 31)

- [Revisiting the Platonic Representation Hypothesis: An Aristotelian View](https://mlsanity.com/paper/2602.14486/) (2026)
- [Back into Plato's Cave: Examining Cross-modal Representational Convergence at Scale](https://mlsanity.com/paper/2604.18572/) (2026)
- [To Align or Not to Align: Strategic Multimodal Representation Alignment for Optimal Performance](https://mlsanity.com/paper/2511.12121/) (2025)
- [Neural Thermodynamics: Entropic Forces in Deep and Universal Representation Learning](https://mlsanity.com/paper/2505.12387/) (2025)
- [Proof of a perfect platonic representation hypothesis](https://mlsanity.com/paper/2507.01098/) (2025)
- [MOVER: Multimodal Optimal Transport with Volume-based Embedding Regularization](https://mlsanity.com/paper/2508.12149/) (2025)
- [Dynamic Reflections: Probing Video Representations with Text Alignment](https://mlsanity.com/paper/2511.02767/) (2025)
- [Multi-modal Data Spectrum: Multi-modal Datasets are Multi-dimensional](https://mlsanity.com/paper/2509.23499/) (2025)
- [Calibrated Multimodal Representation Learning with Missing Modalities](https://mlsanity.com/paper/2511.12034/) (2025)
- [Towards Ambiguity-Free Spatial Foundation Model: Rethinking and Decoupling Depth Ambiguity](https://mlsanity.com/paper/2503.06014/) (2025)
- [Cross-Model Semantics in Representation Learning](https://mlsanity.com/paper/2508.03649/) (2025)
- [Unique Lives, Shared World: Learning from Single-Life Videos](https://mlsanity.com/paper/2512.04085/) (2025)
- [Time Series, Vision, and Language: Exploring the Limits of Alignment in Contrastive Representation Spaces](https://mlsanity.com/paper/2602.19367/) (2026)
- [EmergentBridge: Improving Zero-Shot Cross-Modal Transfer in Unified Multimodal Embedding Models](https://mlsanity.com/paper/2604.11043/) (2026)
- [Characterizing Universal Object Representations Across Vision Models](https://mlsanity.com/paper/2605.13675/) (2026)
- [FairSSL: Fair Multimodal Self-Supervised Learning](https://mlsanity.com/paper/2508.16748/) (2025)
- [What Converges in the Platonic Representation Hypothesis? Structure over Geometry](https://mlsanity.com/paper/2609.27252/) (2026)
- [Beyond Accuracy: Metrics that Uncover What Makes a 'Good' Visual Descriptor](https://mlsanity.com/paper/2507.03542/) (2025)
- [Aligning by Misaligning: Boundary-aware Curriculum Learning for Multimodal Alignment](https://mlsanity.com/paper/2511.08399/) (2025)
- [A framework for auditing grounding claims](https://mlsanity.com/paper/2512.06205/) (2025)
- [A Vision for Multisensory Intelligence: Sensing, Science, and Synergy](https://mlsanity.com/paper/2601.04563/) (2026)
- [The Wittgensteinian Representation Hypothesis: Is Language the Attractor of Multimodal Convergence?](https://mlsanity.com/paper/2605.09352/) (2026)
- [CodeBind: Decoupled Representation Learning for Multimodal Alignment with Unified Compositional Codebook](https://mlsanity.com/paper/2605.18257/) (2026)
- [RePercENT: Scaling Disentangled Representation Learning Beyond Two Modalities](https://mlsanity.com/paper/2606.05109/) (2026)
- [Learning Deep Modality-Shared Self-Expressiveness for Image Clustering with Textual Information](https://mlsanity.com/paper/2608.08418/) (2026)
- [Hub-Spectral Activation of Latent Multimodal Knowledge](https://mlsanity.com/paper/2609.17094/) (2026)
- [Displacement Geometry Captures Platonic Shared Reality Across Models and Modalities](https://mlsanity.com/paper/2609.24209/) (2026)
- [The Alignment Illusion in Multimodal Large Language Models](https://mlsanity.com/paper/2609.30210/) (2026)
- [Platonic Task Arithmetic](https://mlsanity.com/paper/2610.00929/) (2026)
- [Shared Geometry Is Not Shared Physics: A Layerwise Test of the Platonic Representation Hypothesis in Astronomy](https://mlsanity.com/paper/2610.04130/) (2026)

## Abstract (arXiv)

Multimodal representation learning is fundamentally about transforming incomparable modalities into comparable representations. While prior research primarily focused on explicitly aligning these representations through targeted learning objectives and model architectures, a recent line of work has found that independently trained unimodal models of increasing scale and performance can become implicitly aligned with each other. These findings raise fundamental questions regarding the emergence of aligned representations in multimodal learning. Specifically: (1) when and why does alignment emerge implicitly? and (2) is alignment a reliable indicator of performance? Through a comprehensive empirical investigation, we demonstrate that both the emergence of alignment and its relationship with task performance depend on several critical data characteristics. These include, but are not necessarily limited to, the degree of similarity between the modalities and the balance between redundant and unique information they provide for the task. Our findings suggest that alignment may not be universally beneficial; rather, its impact on performance varies depending on the dataset and task. These insights can help practitioners determine whether increasing alignment between modalities is advantageous or, in some cases, detrimental to achieving optimal performance. Code is released at https://github.com/MeganTj/multimodal_alignment.