mlsanityData to 7 Oct 2026

DINOv2: Learning Robust Visual Features without Supervision

arXiv 2304.07193 ↗First posted 14 Apr 2023cs.CVcited by 7,764

Not on the map. Why?

Cited by 789 in Diffusion models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

The recent breakthroughs in natural language processing for model pretraining on large quantities of data have opened the way for similar foundation models in computer vision. These models could greatly simplify the use of images in any system by producing all-purpose visual features, i.e., features that work across image distributions and tasks without finetuning. This work shows that existing pretraining methods, especially self-supervised methods, can produce such features if trained on enough curated data from diverse sources. We revisit existing approaches and combine different techniques to scale our pretraining in terms of data and model size. Most of the technical contributions aim at accelerating and stabilizing the training at scale. In terms of data, we propose an automatic pipeline to build a dedicated, diverse, and curated image dataset instead of uncurated data, as typically done in the self-supervised literature. In terms of models, we train a ViT model (Dosovitskiy et al., 2020) with 1B parameters and distill it into a series of smaller models that surpass the best available all-purpose features, OpenCLIP (Ilharco et al., 2021) on most of the benchmarks at image and pixel levels.

Key facts. Cited by 7,764 papers in the map (data through 7 Oct 2026). Builds on 110 of its 141 references that are in the map. First posted on arXiv 14 Apr 2023.

Builds on

110 of 141 references.

References · 110

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  3. 2020Language Models are Few-Shot Learnerscited by 32,199
  4. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  5. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  6. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  7. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 11,979
  8. 2015Distilling the Knowledge in a Neural Networkcited by 10,076
  9. 2014ImageNet Large Scale Visual Recognition Challengecited by 8,441
  10. 2019Momentum Contrast for Unsupervised Visual Representation Learningcited by 7,482
  11. 2021Masked Autoencoders Are Scalable Vision Learnerscited by 7,339
  12. 2021Emerging Properties in Self-Supervised Vision Transformerscited by 6,893
38 more
  1. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
  2. 2020Bootstrap your own latent: A new approach to self-supervised Learningcited by 4,712
  3. 2016The Cityscapes Dataset for Semantic Urban Scene Understandingcited by 4,427
  4. 2022LAION-5B: An open large-scale dataset for training next generation image-text modelscited by 4,131
  5. 2021On the Opportunities and Risks of Foundation Modelscited by 3,961
  6. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,611
  7. 2018Encoder-Decoder with Atrous Separable Convolution for Semantic Image Segmentationcited by 3,210
  8. 2022Training Compute-Optimal Large Language Modelscited by 2,927
  9. 2019Benchmarking Neural Network Robustness to Common Corruptions and Perturbationscited by 2,857
  10. 2020Exploring Simple Siamese Representation Learningcited by 2,853
  11. 2020Unsupervised Learning of Visual Features by Contrasting Cluster Assignmentscited by 2,845
  12. 2017Billion-scale similarity search with GPUscited by 2,477
  13. 2021BEiT: BERT Pre-Training of Image Transformerscited by 2,360
  14. 2021Masked-attention Mask Transformer for Universal Image Segmentationcited by 2,338
  15. 2013Describing Textures in the Wildcited by 2,110
  16. 2017The Kinetics Human Action Video Datasetcited by 1,894
  17. 2020The Many Faces of Robustness: A Critical Analysis of Out-of-Distribution Generalizationcited by 1,842
  18. 2018Unsupervised Feature Learning via Non-Parametric Instance-level Discriminationcited by 1,838
  19. 2021Vision Transformers for Dense Predictioncited by 1,789
  20. 2020GLU Variants Improve Transformercited by 1,724
  21. 2013Fine-Grained Visual Classification of Aircraftcited by 1,701
  22. 2021An Empirical Study of Training Self-Supervised Vision Transformerscited by 1,610
  23. 2018Unsupervised Representation Learning by Predicting Image Rotationscited by 1,586
  24. 2021LAION-400M: Open Dataset of CLIP-Filtered 400 Million Image-Text Pairscited by 1,544
  25. 2019Do ImageNet Classifiers Generalize to ImageNet?cited by 1,507
  26. 2016Context Encoders: Feature Learning by Inpaintingcited by 1,424
  27. 2019Natural Adversarial Examplescited by 1,401
  28. 2016Unsupervised Learning of Visual Representations by Solving Jigsaw Puzzlescited by 1,379
  29. 2022VideoMAE: Masked Autoencoders are Data-Efficient Learners for Self-Supervised Video Pre-Trainingcited by 1,305
  30. 2019Energy and Policy Considerations for Deep Learning in NLPcited by 1,242
  31. 2022Reproducible scaling laws for contrastive language-image learningcited by 1,163
  32. 2016Colorful Image Colorizationcited by 1,113
  33. 2017The "something something" video database for learning and evaluating visual common sensecited by 1,092
  34. 2015Unsupervised Visual Representation Learning by Context Predictioncited by 1,062
  35. 2018Deep Clustering for Unsupervised Learning of Visual Featurescited by 1,020
  36. 2021Scaling Vision Transformerscited by 993
  37. 2023Self-Supervised Learning from Images with a Joint-Embedding Predictive Architecturecited by 982
  38. 2016Deep Networks with Stochastic Depthcited by 973

50 most cited of 110 All 110 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 365; 2024: 1493; 2025: 2805; 2026: 3101.

7,764 papers.

Sort:

By area:

  1. 2024OpenVLA: An Open-Source Vision-Language-Action Modelcited by 2,777
  2. 2023InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Taskscited by 2,534
  3. 2023DUSt3R: Geometric 3D Vision Made Easycited by 1,526
  4. 2025VGGT: Visual Geometry Grounded Transformercited by 1,475
  5. 2024Depth Anything V2cited by 1,385
  6. 2024Depth Anything: Unleashing the Power of Large-Scale Unlabeled Datacited by 1,346
  7. 2024How Far Are We to GPT-4V? Closing the Gap to Commercial Multimodal Models with Open-Source Suitescited by 1,033
  8. 2024Visual Autoregressive Modeling: Scalable Image Generation via Next-Scale Predictioncited by 1,005
  9. 2024Grounding Image Matching in 3D with MASt3Rcited by 960
  10. 2023A Survey on Multimodal Large Language Modelscited by 898
  11. 2023Vision Transformers Need Registerscited by 844
  12. 2025Fine-Tuning Vision-Language-Action Models: Optimizing Speed and Successcited by 844
38 more
  1. 2024Cambrian-1: A Fully Open, Vision-Centric Exploration of Multimodal LLMscited by 822
  2. 2024Structured 3D Latents for Scalable and Versatile 3D Generationcited by 742
  3. 2023Adversarial Diffusion Distillationcited by 741
  4. 2025V-JEPA 2: Self-Supervised Video Models Enable Understanding, Prediction and Planningcited by 738
  5. 2024Eyes Wide Shut? Exploring the Visual Shortcomings of Multimodal LLMscited by 719
  6. 2024Genie: Generative Interactive Environmentscited by 706
  7. 2024Representation Alignment for Generation: Training Diffusion Transformers Is Easier Than You Thinkcited by 666
  8. 2025Depth Anything 3: Recovering the Visual Space from Any Viewscited by 626
  9. 2024Thinking in Space: How Multimodal Large Language Models See, Remember, and Recall Spacescited by 580
  10. 2024LLaVA-NeXT-Interleave: Tackling Multi-image, Video, and 3D in Large Multimodal Modelscited by 571
  11. 2023CoTracker: It is Better to Track Togethercited by 516
  12. 2024Revisiting Feature Prediction for Learning Visual Representations from Videocited by 490
  13. 2023End-to-end Autonomous Driving: Challenges and Frontierscited by 485
  14. 2024CLAY: A Controllable Large-scale Generative Model for Creating High-quality 3D Assetscited by 444
  15. 2023AnyDoor: Zero-shot Object-level Image Customizationcited by 411
  16. 2024CogACT: A Foundational Vision-Language-Action Model for Synergizing Cognition and Action in Robotic Manipulationcited by 411
  17. 2023Analyzing and Improving the Training Dynamics of Diffusion Modelscited by 396
  18. 2024Depth Pro: Sharp Monocular Metric Depth in Less Than a Secondcited by 395
  19. 2024Metric3Dv2: A Versatile Monocular Geometric Foundation Model for Zero-shot Metric Depth and Surface Normal Estimationcited by 389
  20. 2025UniVLA: Learning to Act Anywhere with Task-centric Latent Actionscited by 386
  21. 2024The Faiss librarycited by 370
  22. 2024SV3D: Novel Multi-view Synthesis and 3D Generation from a Single Image using Latent Video Diffusioncited by 364
  23. 2023ConceptGraphs: Open-Vocabulary 3D Scene Graphs for Perception and Planningcited by 357
  24. 2024LongVU: Spatiotemporal Adaptive Compression for Long Video-Language Understandingcited by 341
  25. 2024Prismatic VLMs: Investigating the Design Space of Visually-Conditioned Language Modelscited by 336
  26. 2024Molmo and PixMo: Open Weights and Open Data for State-of-the-Art Vision-Language Modelscited by 321
  27. 2025Perception Encoder: The best visual embeddings are not at the output of the networkcited by 315
  28. 2024InternVideo2: Scaling Foundation Models for Multimodal Video Understandingcited by 310
  29. 2024ReKep: Spatio-Temporal Reasoning of Relational Keypoint Constraints for Robotic Manipulationcited by 305
  30. 2023Unified-IO 2: Scaling Autoregressive Multimodal Models with Vision, Language, Audio, and Actioncited by 301
  31. 2024TraceVLA: Visual Trace Prompting Enhances Spatial-Temporal Awareness for Generalist Robotic Policiescited by 298
  32. 2023A Tale of Two Features: Stable Diffusion Complements DINO for Zero-Shot Semantic Correspondencecited by 297
  33. 2023RoMa: Robust Dense Feature Matchingcited by 296
  34. 2025MapAnything: Universal Feed-Forward Metric 3D Reconstructioncited by 295
  35. 2025Layer by Layer: Uncovering Hidden Representations in Language Modelscited by 293
  36. 2025π3: Permutation-Equivariant Visual Geometry Learningcited by 292
  37. 2024MoGe: Unlocking Accurate Monocular Geometry Estimation for Open-Domain Images with Optimal Training Supervisioncited by 290
  38. 2023OpenMask3D: Open-Vocabulary 3D Instance Segmentationcited by 288

50 most cited of 7,764 All 7,764 (JSON)