mlsanityData to 14 Oct 2026

DINOv2 Meets Text: A Unified Framework for Image- and Pixel-Level Vision-Language Alignment

arXiv 2412.16334 ↗First posted 20 Dec 2024cs.CVcited by 79

Not on the map. Why?

Abstract (arXiv)

Self-supervised visual foundation models produce powerful embeddings that achieve remarkable performance on a wide range of downstream tasks. However, unlike vision-language models such as CLIP, self-supervised visual features are not readily aligned with language, hindering their adoption in open-vocabulary tasks. Our method, named dino.txt, unlocks this new ability for DINOv2, a widely used self-supervised visual encoder. We build upon the LiT training strategy, which trains a text encoder to align with a frozen vision model but leads to unsatisfactory results on dense tasks. We propose several key ingredients to improve performance on both global and dense tasks, such as concatenating the [CLS] token with the patch average to train the alignment and curating data using both text and image modalities. With these, we successfully train a CLIP-like model with only a fraction of the computational cost compared to CLIP while achieving state-of-the-art results in zero-shot classification and open-vocabulary semantic segmentation.

Key facts. Cited by 79 papers in the map (data through 14 Oct 2026). Builds on 87 of its 108 references that are in the map. First posted on arXiv 20 Dec 2024.

Builds on

87 of 108 references.

References · 87

  1. 2017Attention Is All You Needcited by 60,361
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  3. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  4. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  5. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,305
  6. 2023DINOv2: Learning Robust Visual Features without Supervisioncited by 7,974
  7. 2023Visual Instruction Tuningcited by 7,869
  8. 2018Representation Learning with Contrastive Predictive Codingcited by 7,777
  9. 2019Momentum Contrast for Unsupervised Visual Representation Learningcited by 7,513
  10. 2021Masked Autoencoders Are Scalable Vision Learnerscited by 7,403
  11. 2021Emerging Properties in Self-Supervised Vision Transformerscited by 6,968
  12. 2020Bootstrap your own latent: A new approach to self-supervised Learningcited by 4,750
38 more
  1. 2016The Cityscapes Dataset for Semantic Urban Scene Understandingcited by 4,444
  2. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,386
  3. 2022LAION-5B: An open large-scale dataset for training next generation image-text modelscited by 4,152
  4. 2023Sigmoid Loss for Language Image Pre-Trainingcited by 3,109
  5. 2020Unsupervised Learning of Visual Features by Contrasting Cluster Assignmentscited by 2,857
  6. 2023InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Taskscited by 2,565
  7. 2021BEiT: BERT Pre-Training of Image Transformerscited by 2,375
  8. 2021An Empirical Study of Training Self-Supervised Vision Transformerscited by 1,616
  9. 2024Expanding Performance Boundaries of Open-Source Multimodal Models with Model, Data, and Test-Time Scalingcited by 1,599
  10. 2021LAION-400M: Open Dataset of CLIP-Filtered 400 Million Image-Text Pairscited by 1,555
  11. 2019Do ImageNet Classifiers Generalize to ImageNet?cited by 1,515
  12. 2019Natural Adversarial Examplescited by 1,408
  13. 2015Flickr30k Entities: Collecting Region-to-Phrase Correspondences for Richer Image-to-Sentence Modelscited by 1,362
  14. 2021Conceptual 12M: Pushing Web-Scale Image-Text Pre-Training To Recognize Long-Tail Visual Conceptscited by 1,216
  15. 2022Reproducible scaling laws for contrastive language-image learningcited by 1,173
  16. 2016Semantic Understanding of Scenes through the ADE20K Datasetcited by 1,069
  17. 2024How Far Are We to GPT-4V? Closing the Gap to Commercial Multimodal Models with Open-Source Suitescited by 1,050
  18. 2023Self-Supervised Learning from Images with a Joint-Embedding Predictive Architecturecited by 1,039
  19. 2023Vision Transformers Need Registerscited by 871
  20. 2016COCO-Stuff: Thing and Stuff Classes in Contextcited by 864
  21. 2024Cambrian-1: A Fully Open, Vision-Centric Exploration of Multimodal LLMscited by 838
  22. 2023EVA-CLIP: Improved Training Techniques for CLIP at Scalecited by 798
  23. 2022Language-driven Semantic Segmentationcited by 717
  24. 2019Large-Scale Long-Tailed Recognition in an Open Worldcited by 705
  25. 2023DataComp: In search of the next generation of multimodal datasetscited by 644
  26. 2022MTEB: Massive Text Embedding Benchmarkcited by 640
  27. 2021RegionCLIP: Region-based Language-Image Pretrainingcited by 616
  28. 2021Extract Free Dense Labels from CLIPcited by 583
  29. 2021LiT: Zero-Shot Transfer with Locked-image text Tuningcited by 577
  30. 2022Open-Vocabulary Semantic Segmentation with Mask-adapted CLIPcited by 565
  31. 2021DenseCLIP: Language-Guided Dense Prediction with Context-Aware Promptingcited by 532
  32. 2022GroupViT: Semantic Segmentation Emerges from Text Supervisioncited by 531
  33. 2021Scaling Open-Vocabulary Image Segmentation with Image-Level Labelscited by 473
  34. 2023ConceptFusion: Open-set Multimodal 3D Mappingcited by 304
  35. 2023SemDeDup: Data-efficient learning at web-scale through semantic deduplicationcited by 303
  36. 2023RoMa: Robust Dense Feature Matchingcited by 302
  37. 2023MIMIC-IT: Multi-Modal In-Context Instruction Tuningcited by 267
  38. 2023Data Filtering Networkscited by 250

50 most cited of 87 All 87 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 29; 2026: 50.

79 papers.

Sort:

By area:

  1. 2025VL-JEPA: Joint Embedding Predictive Architecture for Vision-languagecited by 68
  2. 2025Scaling Language-Free Visual Representation Learningcited by 66
  3. 2025JAFAR: Jack up Any Feature at Any Resolutioncited by 21
  4. 2025Pushing the Frontier of Audiovisual Perception with Large-Scale Multimodal Correspondence Learningcited by 13
  5. 2026TIPSv2: Advancing Vision-Language Pretraining with Enhanced Patch-Text Alignmentcited by 13
  6. 2025CTRL-O: Language-Controllable Object-Centric Visual Representation Learningcited by 12
  7. 2026OneVision-Encoder: Codec-Aligned Sparsity as a Foundational Principle for Multimodal Intelligencecited by 12
  8. 2025Visual Instruction Pretraining for Domain-Specific Foundation Modelscited by 9
  9. 2025Towards Deploying VLA without Fine-Tuning: Plug-and-Play Inference-Time VLA Policy Steering via Embodied Evolutionary Diffusioncited by 8
  10. 2025UniVCD: A New Method for Unsupervised Change Detection in the Open-Vocabulary Eracited by 7
  11. 2025ConceptPose: Training-Free Zero-Shot Object Pose Estimation using Concept Vectorscited by 5
  12. 2026DynaFLIP: Rethinking Robotics Perception via Tri-Modal-Dynamics Guided Representationcited by 5
38 more
  1. 2025RadZero: Similarity-Based Cross-Attention for Explainable Vision-Language Alignment in Chest X-ray with Zero-Shot Multi-Task Capabilitycited by 4
  2. 2025Generalizable Object Re-Identification via Visual In-Context Promptingcited by 4
  3. 2026Taming SAM3 in the Wild: A Concept Bank for Open-Vocabulary Segmentationcited by 4
  4. 2026Open-Vocabulary Domain Generalization in Urban-Scene Segmentationcited by 4
  5. 2025SigLino: Efficient Multi-Teacher Distillation for Agglomerative Vision Foundation Modelscited by 3
  6. 2026PEARL: Geometry Aligns Semantics for Training-Free Open-Vocabulary Semantic Segmentationcited by 3
  7. 2026Learning Robust Visual Features in Computed Tomography Enables Efficient Transfer Learning for Clinical Taskscited by 3
  8. 2025ThematicPlane: Bridging Tacit User Intent and Latent Spaces for Image Generationcited by 2
  9. 2025One Patch to Caption Them All: A Unified Zero-Shot Captioning Frameworkcited by 2
  10. 2025GRPO-RM: Fine-Tuning Representation Models via GRPO-Driven Reinforcement Learningcited by 2
  11. 2026SpatialBoost: Enhancing Visual Representation through Language-Guided Reasoningcited by 2
  12. 2026Streamlined Open-Vocabulary Human-Object Interaction Detectioncited by 2
  13. 2026T-REN: Learning Text-Aligned Region Tokens Improves Dense Vision-Language Alignment and Scalabilitycited by 2
  14. 2026VIP: Visual-guided Prompt Evolution for Efficient Dense Vision-Language Inferencecited by 2
  15. 2026Sparse Code Uplifting for Efficient 3D Language Gaussian Splattingcited by 2
  16. 2025PosBridge: Multi-View Positional Embedding Transplant for Identity-Aware Image Editingcited by 1
  17. 2025Disentangling Latent Embeddings with Sparse Linear Concept Subspaces (SLiCS)cited by 1
  18. 2025Video Consistency Distance: Enhancing Temporal Consistency for Image-to-Video Generation via Reward-Based Fine-Tuningcited by 1
  19. 2025D-PerceptCT: Deep Perceptual Enhancement for Low-Dose CT Imagescited by 1
  20. 2025PowerCLIP: Powerset Alignment for Contrastive Pre-Trainingcited by 1
  21. 2026Granulon: Awakening Pixel-Level Visual Encoders with Adaptive Multi-Granularity Semantics for MLLMcited by 1
  22. 2026ReCap: Lightweight Referential Grounding for Coherent Story Visualizationcited by 1
  23. 2026DINO Soars: DINOv3 for Open-Vocabulary Semantic Segmentation of Remote Sensing Imagerycited by 1
  24. 2026Training-free Cross-domain Few-shot Segmentation via Robust Semantic Representation and Matchingcited by 1
  25. 2026EditVerse3D: High-Quality 3D Object Editing with Region-Aware Learningcited by 1
  26. 2026DINOde: Continuous Vision-Text Alignment for Open-Vocabulary Semantic Segmentationcited by 1
  27. 2026Standalone DINOv3 for Training-Free Open-Vocabulary Semantic Segmentation in Remote Sensingcited by 1
  28. 2025Memory-Modular Classification: Learning to Generalize with Memory Replacementcited by 0
  29. 2025Automated data curation for self-supervised learning in underwater acoustic analysiscited by 0
  30. 2025Improving Robustness of Foundation Models in Domain Adaptation with Soup-Adapterscited by 0
  31. 2025VeS: Teaching Pixels to Listen Without Supervisioncited by 0
  32. 2025SynSeg: Feature Synergy for Multi-Category Contrastive Learning in End-to-End Open-Vocabulary Semantic Segmentationcited by 0
  33. 2025Make me an Expert: Distilling from Generalist Black-Box Models into Specialized Models for Semantic Segmentationcited by 0
  34. 2025ReCap: Event-Aware Image Captioning with Article Retrieval and Semantic Gaussian Normalizationcited by 0
  35. 2025UniADC: A Unified Framework for Anomaly Detection and Classificationcited by 0
  36. 2025Adapting MLLMs for Nuanced Video Retrievalcited by 0
  37. 2026TRACER: Texture-Robust Affordance Chain-of-Thought for Deformable-Object Refinementcited by 0
  38. 2026Online Monitoring Framework for Automotive Time Series Data using JEPA Embeddingscited by 0

50 most cited of 79 All 79 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024Talking to DINO: Bridging Self-Supervised Vision Backbones with Language for Open-Vocabulary Segmentation
  2. 2025Kernel-based Unsupervised Embedding Alignment for Enhanced Visual Representation in Vision-language Models
  3. 2023TagAlign: Improving Vision-Language Alignment with Multi-Tag Classification
  4. 2024Harnessing Frozen Unimodal Encoders for Flexible Multimodal Alignment
  5. 2022PyramidCLIP: Hierarchical Feature Alignment for Vision-language Model Pretraining
  6. 2024Language-Driven Visual Consensus for Zero-Shot Semantic Segmentation
  7. 2026Same Semantics, Different Paths: Self-Improving Alignment for Vision-Text Compression
  8. 2022Single-Stream Multi-Level Alignment for Vision-Language Pretraining