mlsanityData to 14 Oct 2026

Kernel-based Unsupervised Embedding Alignment for Enhanced Visual Representation in Vision-language Models

arXiv 2506.02557 ↗First posted 3 Jun 2025cs.CVcited by 15

On the map: Multimodal LLMs / vision-language models → MLLM training and visual instruction tuning

Abstract (arXiv)

Vision-language models, such as CLIP, have achieved significant success in aligning visual and textual representations, becoming essential components of many multi-modal large language models (MLLMs) like LLaVA and OpenFlamingo. However, numerous studies have identified CLIP's limited fine-grained perception as a critical drawback, leading to substantial failures in downstream MLLMs. In contrast, vision-centric foundation models like DINOv2 demonstrate remarkable capabilities in capturing fine details from images. In this work, we propose a novel kernel-based method to align CLIP's visual representation with that of DINOv2, ensuring that the resulting embeddings maintain compatibility with text embeddings while enhancing perceptual capabilities. Our alignment objective is designed for efficient stochastic optimization. Following this image-only alignment fine-tuning, the visual encoder retains compatibility with the frozen text encoder and exhibits significant improvements in zero-shot object recognition, fine-grained spatial reasoning, and localization. By integrating the aligned visual encoder, downstream MLLMs also demonstrate enhanced performance.

Key facts. Cited by 15 papers in the map (data through 14 Oct 2026). Placed in MLLM training and visual instruction tuning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 75 of its 97 references that are in the map. First posted on arXiv 3 Jun 2025.

Builds on

75 of 97 references.

Closest ancestors · 4

Same direction, newest first

  1. 2024Eagle: Exploring The Design Space for Multimodal LLMs with Mixture of Encoders28 Aug
  2. 2024Eyes Wide Shut? Exploring the Visual Shortcomings of Multimodal LLMs11 Jan
  3. 2023InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Tasks21 Dec
  4. 2023From CLIP to DINO: Visual Encoders Shout in Multi-modal Large Language Models13 Oct

Other references · 71

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  2. 2014Microsoft COCO: Common Objects in Contextcited by 17,608
  3. 2023DINOv2: Learning Robust Visual Features without Supervisioncited by 7,974
  4. 2023Visual Instruction Tuningcited by 7,869
  5. 2021Masked Autoencoders Are Scalable Vision Learnerscited by 7,403
  6. 2021Emerging Properties in Self-Supervised Vision Transformerscited by 6,968
  7. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,137
  8. 2023Sigmoid Loss for Language Image Pre-Trainingcited by 3,109
  9. 2016Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answeringcited by 2,516
  10. 2021BEiT: BERT Pre-Training of Image Transformerscited by 2,375
  11. 2021Learning to Prompt for Vision-Language Modelscited by 2,281
  12. 2023Qwen-VL: A Versatile Vision-Language Model for Understanding, Localization, Text Reading, and Beyondcited by 2,224
38 more
  1. 2013Describing Textures in the Wildcited by 2,125
  2. 2023Evaluating Object Hallucination in Large Vision-Language Modelscited by 1,729
  3. 2019Towards VQA Models That Can Readcited by 1,718
  4. 2015Microsoft COCO Captions: Data Collection and Evaluation Servercited by 1,462
  5. 2016CLEVR: A Diagnostic Dataset for Compositional Language and Elementary Visual Reasoningcited by 1,457
  6. 2019Natural Adversarial Examplescited by 1,408
  7. 2018Demystifying MMD GANscited by 1,293
  8. 2019OK-VQA: A Visual Question Answering Benchmark Requiring External Knowledgecited by 1,121
  9. 2016Modeling Context in Referring Expressionscited by 1,063
  10. 2021CLIP-Adapter: Better Vision-Language Models with Feature Adapterscited by 1,026
  11. 2019Learning Robust Global Representations by Penalizing Local Predictive Powercited by 1,013
  12. 2023Vision Transformers Need Registerscited by 871
  13. 2018VizWiz Grand Challenge: Answering Visual Questions from Blind Peoplecited by 858
  14. 2016A Diagram Is Worth A Dozen Imagescited by 804
  15. 2023EVA-CLIP: Improved Training Techniques for CLIP at Scalecited by 798
  16. 2017Remote Sensing Image Scene Classification: Benchmark and State of the Artcited by 797
  17. 2023DataComp: In search of the next generation of multimodal datasetscited by 644
  18. 2021RegionCLIP: Region-based Language-Image Pretrainingcited by 616
  19. 2022When and why vision-language models behave like bags-of-words, and what to do about it?cited by 610
  20. 2023OpenFlamingo: An Open-Source Framework for Training Large Autoregressive Vision-Language Modelscited by 565
  21. 2022Winoground: Probing Vision and Language Models for Visio-Linguistic Compositionalitycited by 529
  22. 2020The Hateful Memes Challenge: Detecting Hate Speech in Multimodal Memescited by 490
  23. 2018Rotation Equivariant CNNs for Digital Pathologycited by 422
  24. 2022Tip-Adapter: Training-free Adaption of CLIP for Few-shot Classificationcited by 362
  25. 2022The Vendi Score: A Diversity Evaluation Metric for Machine Learningcited by 311
  26. 2013Challenges in Representation Learning: A report on three machine learning contestscited by 302
  27. 2020Towards Understanding Ensemble, Knowledge Distillation and Self-Distillation in Deep Learningcited by 297
  28. 2022Visual Spatial Reasoningcited by 293
  29. 2023Demystifying CLIP Datacited by 270
  30. 2023What's "up" with vision-language models? Investigating their struggle with spatial reasoningcited by 254
  31. 2023Data Filtering Networkscited by 250
  32. 2018Explicit Inductive Bias for Transfer Learning with Convolutional Networkscited by 227
  33. 2023Exposing flaws of generative model evaluation metrics and their unfair treatment of diffusion modelscited by 205
  34. 2023Convolutions Die Hard: Open-Vocabulary Segmentation with Single Frozen Convolutional CLIPcited by 201
  35. 2024LLaVA-UHD: an LMM Perceiving Any Aspect Ratio and High-Resolution Imagescited by 195
  36. 2023AM-RADIO: Agglomerative Vision Foundation Model -- Reduce All Domains Into Onecited by 169
  37. 2018TallyQA: Answering Complex Counting Questionscited by 153
  38. 2024Robust CLIP: Unsupervised Adversarial Fine-Tuning of Vision Embeddings for Robust Large Vision-Language Modelscited by 120

50 most cited of 71 All 71 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 3; 2026: 12.

15 papers.

Sort:

By area:

  1. 2025SPARKE: Scalable Prompt-Aware Diversity and Novelty Guidance in Diffusion Models via RKE Scorecited by 14
  2. 2025Towards an Explainable Comparison and Alignment of Feature Embeddingscited by 10
  3. 2026The Maximum von Neumann Entropy Principle: Theory and Applications in Machine Learningcited by 3
  4. 2025Theoretical Refinement of CLIP by Utilizing Linear Structure of Optimal Similaritycited by 2
  5. 2026DAK-UCB: Diversity-Aware Prompt Routing for LLMs and Generative Modelscited by 2
  6. 2026UniCon: Unified Framework for Efficient Contrastive Alignment via Kernelscited by 2
  7. 2026Diverse via bounded Agreement: Geometric Regularization for Multimodal Fusioncited by 1
  8. 2026Seeing Clearly, Reasoning Confidently: Plug-and-Play Remedies for Vision Language Model Blindnesscited by 1
  9. 2026Mixture-Greedy for Online Generative Model Selection: Is UCB Necessary in Diversity-Aware Multi-Armed Bandits?cited by 1
  10. 2026KALE: Kernel Alignment with Loss Equilibration for Stable CLIP-DINOv2 Alignment at Web Scalecited by 1
  11. 2026A3B2: Adaptive Asymmetric Adapter for Alleviating Branch Bias in Vision-Language Image Classification with Few-Shot Learningcited by 0
  12. 2026Kernel-Based Steering of CLIP with Vision-Language Model Preferencescited by 0
3 more
  1. 2026Seeing Is Not Addressing: Auditing Linguistic Access to Frozen Visual Geometrycited by 0
  2. 2026Revisiting Visual Representation Enhancement of VLMs via Kernel Canonical Correlation Analysiscited by 0
  3. 2026Unlocking Fine-Grained Perception in CLIP via Structurally-Aware Latent Masked Modelingcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024DINOv2 Meets Text: A Unified Framework for Image- and Pixel-Level Vision-Language Alignment
  2. 2025ProCLIP: Progressive Vision-Language Alignment via LLM-based Embedder
  3. 2025BASIC: Boosting Visual Alignment with Intrinsic Refined Embeddings in Multimodal Large Language Models
  4. 2022X2-VLM: All-In-One Pre-trained Model For Vision-Language Tasks
  5. 2025Visual Representation Alignment for Multimodal Large Language Models
  6. 2025Analyzing Fine-Grained Alignment and Enhancing Vision Understanding in Multimodal Language Models
  7. 2025LangBridge: Interpreting Image as a Combination of Language Embeddings
  8. 2025FineLIP: Extending CLIP's Reach via Fine-Grained Alignment with Longer Text Inputs