mlsanityData to 7 Oct 2026

Harnessing Frozen Unimodal Encoders for Flexible Multimodal Alignment

arXiv 2409.19425 ↗First posted 28 Sep 2024cs.CVcited by 11

On the map: Multimodal LLMs / vision-language models → MLLM training and visual instruction tuning

Abstract (arXiv)

Recent contrastive multimodal vision-language models like CLIP have demonstrated robust open-world semantic understanding, becoming the standard image backbones for vision-language applications. However, recent findings suggest high semantic similarity between well-trained unimodal encoders, which raises a key question: Is there a plausible way to connect unimodal backbones for vision-language tasks? To this end, we propose a novel framework that aligns vision and language using frozen unimodal encoders. It involves selecting semantically similar encoders in the latent space, curating a concept-rich dataset of image-caption pairs, and training simple MLP projectors. We evaluated our approach on 12 zero-shot classification datasets and 2 image-text retrieval datasets. Our best model, utilizing DINOv2 and All-Roberta-Large text encoder, achieves 76(%) accuracy on ImageNet with a 20-fold reduction in data and 65-fold reduction in compute requirements compared multi-modal alignment where models are trained from scratch. The proposed framework enhances the accessibility of multimodal model development while enabling flexible adaptation across diverse scenarios. Code and curated datasets are available at textttgithub.com/mayug/freeze-align.

Key facts. Cited by 11 papers in the map (data through 7 Oct 2026). Placed in MLLM training and visual instruction tuning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 43 of its 64 references that are in the map. First posted on arXiv 28 Sep 2024.

Builds on

43 of 64 references.

Closest ancestors · 2

Same direction, newest first

  1. 2024Cambrian-1: A Fully Open, Vision-Centric Exploration of Multimodal LLMs24 Jun
  2. 2023ShareGPT4V: Improving Large Multi-Modal Models with Better Captions21 Nov

Other references · 41

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  3. 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,049
  4. 2023DINOv2: Learning Robust Visual Features without Supervisioncited by 7,764
  5. 2018Representation Learning with Contrastive Predictive Codingcited by 7,662
  6. 2021Emerging Properties in Self-Supervised Vision Transformerscited by 6,893
  7. 2020Training data-efficient image transformers & distillation through attentioncited by 4,560
  8. 2022LAION-5B: An open large-scale dataset for training next generation image-text modelscited by 4,131
  9. 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,848
  10. 2023Sigmoid Loss for Language Image Pre-Trainingcited by 3,048
  11. 2021SimCSE: Simple Contrastive Learning of Sentence Embeddingscited by 2,168
  12. 2013Fine-Grained Visual Classification of Aircraftcited by 1,701
29 more
  1. 2019Similarity of Neural Network Representations Revisitedcited by 1,695
  2. 2021LAION-400M: Open Dataset of CLIP-Filtered 400 Million Image-Text Pairscited by 1,544
  3. 2017EuroSAT: A Novel Dataset and Deep Learning Benchmark for Land Use and Land Cover Classificationcited by 1,488
  4. 2015Flickr30k Entities: Collecting Region-to-Phrase Correspondences for Richer Image-to-Sentence Modelscited by 1,352
  5. 2022No Language Left Behind: Scaling Human-Centered Machine Translationcited by 1,290
  6. 2021BitFit: Simple Parameter-efficient Fine-tuning for Transformer-based Masked Language-modelscited by 1,234
  7. 2021Conceptual 12M: Pushing Web-Scale Image-Text Pre-Training To Recognize Long-Tail Visual Conceptscited by 1,209
  8. 2021iBOT: Image BERT Pre-Training with Online Tokenizercited by 891
  9. 2017Remote Sensing Image Scene Classification: Benchmark and State of the Artcited by 791
  10. 2023DataComp: In search of the next generation of multimodal datasetscited by 635
  11. 2022MTEB: Massive Text Embedding Benchmarkcited by 623
  12. 2021Extract Free Dense Labels from CLIPcited by 580
  13. 2021LiT: Zero-Shot Transfer with Locked-image text Tuningcited by 572
  14. 2024The Platonic Representation Hypothesiscited by 362
  15. 2014Understanding image representations by measuring their equivariance and equivalencecited by 240
  16. 2023Improving CLIP Training with Language Rewritescited by 216
  17. 2021Revisiting Model Stitching to Compare Neural Representationscited by 203
  18. 2022Relative representations enable zero-shot latent space communicationcited by 175
  19. 2022Linearly Mapping from Image to Text Spacecited by 139
  20. 2022Open Vocabulary Semantic Segmentation with Patch Aligned Contrastive Learningcited by 115
  21. 2023A Picture is Worth More Than 77 Text Tokens: Evaluating CLIP-Style Models on Dense Captionscited by 107
  22. 2023Improving Multimodal Datasets with Image Captioningcited by 93
  23. 2019The HSIC Bottleneck: Deep Learning without Back-Propagationcited by 77
  24. 2023Enhancing CLIP with GPT-4: Harnessing Visual Descriptions as Promptscited by 48
  25. 2024Do Vision and Language Encoders Represent the World Similarly?cited by 44
  26. 2024Improving fine-grained understanding in image-text pre-trainingcited by 44
  27. 2020Towards Zero-shot Cross-lingual Image Retrievalcited by 25
  28. 2023NLLB-CLIP -- train performant multilingual image retrieval model on a budgetcited by 24
  29. 2023Contrastive Alignment of Vision to Language Through Parameter-Efficient Transfer Learningcited by 16

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 3; 2026: 8.

11 papers.

  1. 2025With Limited Data for Multimodal Alignment, Let the STRUCTURE Guide Youcited by 13
  2. 2026SOTAlign: Semi-Supervised Alignment of Unimodal Vision and Language Models via Optimal Transportcited by 2
  3. 2025BotaCLIP: Contrastive Learning for Botany-Aware Representation of Earth Observation Datacited by 1
  4. 2026Enhance-then-Balance Modality Collaboration for Robust Multimodal Sentiment Analysiscited by 1
  5. 2026A Unifying Framework for Concept-Based Representational Similaritycited by 1
  6. 2025MODIS: Multi-Omics Data Integration for Small and unpaired datasetscited by 0
  7. 2026Learning Relative Representations for Fine-Grained Multimodal Alignment with Limited Datacited by 0
  8. 2026Zero-Shot Face-to-Speech Synthesis via Latent Space Adaptation of a Style-Diffusion TTS Modelcited by 0
  9. 2026Multimodal Alignment Through Joint Kernel Entropic Gromov--Wasserstein Optimal Transportcited by 0
  10. 2026Aligning Biomedical Texts and Knowledge Graphs: A Systematic Comparison of Lightweight Alignment Strategiescited by 0
  11. 2026Hub-Spectral Activation of Latent Multimodal Knowledgecited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Assessing and Learning Alignment of Unimodal Vision and Language Models
  2. 2025uCLIP: Parameter-Efficient Multilingual Extension of Vision-Language Models with Unpaired Data
  3. 2026Hyperdimensional Cross-Modal Alignment of Frozen Language and Image Models for Efficient Image Captioning
  4. 2025Analyzing Fine-Grained Alignment and Enhancing Vision Understanding in Multimodal Language Models
  5. 2022Multimodal Adaptive Distillation for Leveraging Unimodal Encoders for Vision-Language Tasks
  6. 2024DINOv2 Meets Text: A Unified Framework for Image- and Pixel-Level Vision-Language Alignment
  7. 2022Single-Stream Multi-Level Alignment for Vision-Language Pretraining
  8. 2024Better Language Models Exhibit Higher Visual Alignment