mlsanityData to 13 Oct 2026

Aligning Multimodal Representations through an Information Bottleneck

arXiv 2506.04870 ↗First posted 5 Jun 2025cs.LGcited by 13

Not on the map. Why?

Abstract (arXiv)

Contrastive losses have been extensively used as a tool for multimodal representation learning. However, it has been empirically observed that their use is not effective to learn an aligned representation space. In this paper, we argue that this phenomenon is caused by the presence of modality-specific information in the representation space. Although some of the most widely used contrastive losses maximize the mutual information between representations of both modalities, they are not designed to remove the modality-specific information. We give a theoretical description of this problem through the lens of the Information Bottleneck Principle. We also empirically analyze how different hyperparameters affect the emergence of this phenomenon in a controlled experimental setup. Finally, we propose a regularization term in the loss function that is derived by means of a variational approximation and aims to increase the representational alignment. We analyze in a set of controlled experiments and real-world applications the advantages of including this regularization term.

Key facts. Cited by 13 papers in the map (data through 13 Oct 2026). Builds on 50 of its 75 references that are in the map. First posted on arXiv 5 Jun 2025.

Builds on

50 of 75 references.

References · 50

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,950
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,588
  3. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,043
  4. 2014Microsoft COCO: Common Objects in Contextcited by 17,550
  5. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 12,009
  6. 2018Representation Learning with Contrastive Predictive Codingcited by 7,712
  7. 2022Hierarchical Text-Conditional Image Generation with CLIP Latentscited by 6,564
  8. 2020Scaling Laws for Neural Language Modelscited by 6,494
  9. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,103
  10. 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generationcited by 4,314
  11. 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,855
  12. 2015FaceNet: A Unified Embedding for Face Recognition and Clusteringcited by 2,838
38 more
  1. 2018Unsupervised Feature Learning via Non-Parametric Instance-level Discriminationcited by 1,841
  2. 2019Similarity of Neural Network Representations Revisitedcited by 1,719
  3. 2021Align before Fuse: Vision and Language Representation Learning with Momentum Distillationcited by 1,616
  4. 2020Understanding Contrastive Representation Learning through Alignment and Uniformity on the Hyperspherecited by 1,397
  5. 2019Contrastive Multiview Codingcited by 1,297
  6. 2018Learning deep representations by mutual information estimation and maximizationcited by 1,269
  7. 2023ImageBind: One Embedding Space To Bind Them Allcited by 1,168
  8. 2014Deep Visual-Semantic Alignments for Generating Image Descriptionscited by 967
  9. 2016Deep Variational Information Bottleneckcited by 924
  10. 2020What Makes for Good Views for Contrastive Learning?cited by 844
  11. 2015Deep Learning and the Information Bottleneck Principlecited by 796
  12. 2019Learning Representations by Maximizing Mutual Information Across Viewscited by 733
  13. 2022EVA: Exploring the Limits of Masked Visual Representation Learning at Scalecited by 689
  14. 2021Do Vision Transformers See Like Convolutional Neural Networks?cited by 674
  15. 2019Data-Efficient Image Recognition with Contrastive Predictive Codingcited by 673
  16. 2021VideoCLIP: Contrastive Pre-training for Zero-shot Video-Text Understandingcited by 600
  17. 2022Mind the Gap: Understanding the Modality Gap in Multi-modal Contrastive Representation Learningcited by 555
  18. 2017Opening the Black Box of Deep Neural Networks via Informationcited by 497
  19. 2020Understanding the Behaviour of Contrastive Losscited by 397
  20. 2024The Platonic Representation Hypothesiscited by 369
  21. 2017Emergence of Invariance and Disentanglement in Deep Representationscited by 213
  22. 2021Revisiting Model Stitching to Compare Neural Representationscited by 204
  23. 2018An efficient framework for learning sentence representationscited by 197
  24. 2020Learning Robust Representations via Multi-View Information Bottleneckcited by 190
  25. 2020A Theory of Usable Information Under Computational Constraintscited by 156
  26. 2016Information Dropout: Learning Optimal Representations Through Noisy Computationcited by 132
  27. 2023Similarity of Neural Network Models: A Survey of Functional and Representational Measurescited by 128
  28. 2021Towards artificial general intelligence via a multimodal foundation modelcited by 112
  29. 2022Disentangled Representation Learningcited by 112
  30. 2019Multimodal End-to-End Autonomous Drivingcited by 110
  31. 2019On the Transfer of Inductive Bias from Simulation to the Real World: a New Disentanglement Datasetcited by 100
  32. 2022Rethinking Minimal Sufficient Representation in Contrastive Learningcited by 68
  33. 2016Inductive Bias of Deep Convolutional Networks through Pooling Geometrycited by 53
  34. 2014Deep Neural Networks Rival the Representation of Primate IT Cortex for Core Visual Object Recognitioncited by 50
  35. 2024Scaling Properties of Speech Language Modelscited by 31
  36. 2024It's Not a Modality Gap: Characterizing and Addressing the Contrastive Gapcited by 26
  37. 2023Theoretical Analysis of Inductive Biases in Deep Convolutional Networkscited by 21
  38. 2023Multimodality of AI for Education: Towards Artificial General Intelligencecited by 21

Built on by

Cited by year (2026 to 13 Oct)

Citing papers per year: 2025: 4; 2026: 9.

13 papers.

Sort:

By area:

  1. 2025ECHO: Toward Contextual Seq2Seq Paradigms in Large EEG Modelscited by 6
  2. 2026LaME: Learning to Think in Latent Space for Multimodal Embedding via Information Bottleneckcited by 5
  3. 2025CrossJEPA: Cross-Modal Joint-Embedding Predictive Architecture for Efficient 3D Representation Learning from 2D Imagescited by 4
  4. 2026A Comprehensive Information-Decomposition Analysis of Large Vision-Language Modelscited by 4
  5. 2025The Describe-Then-Generate Bottleneck: How VLM Descriptions Alter Image Generation Outcomescited by 1
  6. 2026Bridging Functional and Representational Similarity via Usable Informationcited by 1
  7. 2025FDRMFL: Multimodal Federated Feature Extraction Model Based on Information Maximization and Contrastive Learningcited by 0
  8. 2026Toward Structural Multimodal Representations: Specialization, Selection, and Sparsification via Mixture-of-Expertscited by 0
  9. 2026Learning Emergent Modular Representations in Multi-modality Medical Vision Foundation Modelscited by 0
  10. 2026Information-theoretic Multimodal Representation Learning for Electrocardiogram Signalscited by 0
  11. 2026OVA-IB: One vs All Information Bottleneck for Multi-Modal Alignmentcited by 0
  12. 2026SGWIB:Sliced Gromov-Wasserstein Information Bottleneck for Video Highlight Detectioncited by 0
1 more
  1. 2026Architecture-Dependent Fusion Pathways in MLLMscited by 0