# Simplifying DINO via Coding Rate Regularization

arXiv: https://arxiv.org/abs/2502.10385 · HTML: https://mlsanity.com/paper/2502.10385/

## Key facts

- Cited by 21 papers in the map (data through 14 Oct 2026).
- Builds on 34 of its 45 references that are in the map.
- First posted on arXiv 14 Feb 2025.

## Builds on (most cited first, 30 of 34)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [A Simple Framework for Contrastive Learning of Visual Representations](https://mlsanity.com/paper/2002.05709/) (2020)
- [DINOv2: Learning Robust Visual Features without Supervision](https://mlsanity.com/paper/2304.07193/) (2023)
- [Representation Learning with Contrastive Predictive Coding](https://mlsanity.com/paper/1807.03748/) (2018)
- [Momentum Contrast for Unsupervised Visual Representation Learning](https://mlsanity.com/paper/1911.05722/) (2019)
- [Masked Autoencoders Are Scalable Vision Learners](https://mlsanity.com/paper/2111.06377/) (2021)
- [Emerging Properties in Self-Supervised Vision Transformers](https://mlsanity.com/paper/2104.14294/) (2021)
- [Bootstrap your own latent: A new approach to self-supervised Learning](https://mlsanity.com/paper/2006.07733/) (2020)
- [Training data-efficient image transformers & distillation through attention](https://mlsanity.com/paper/2012.12877/) (2020)
- [Exploring Simple Siamese Representation Learning](https://mlsanity.com/paper/2011.10566/) (2020)
- [Unsupervised Learning of Visual Features by Contrasting Cluster Assignments](https://mlsanity.com/paper/2006.09882/) (2020)
- [Sinkhorn Distances: Lightspeed Computation of Optimal Transportation Distances](https://mlsanity.com/paper/1306.0895/) (2013)
- [Unsupervised Feature Learning via Non-Parametric Instance-level Discrimination](https://mlsanity.com/paper/1805.01978/) (2018)
- [VICReg: Variance-Invariance-Covariance Regularization for Self-Supervised Learning](https://mlsanity.com/paper/2105.04906/) (2021)
- [Self-Supervised Learning from Images with a Joint-Embedding Predictive Architecture](https://mlsanity.com/paper/2301.08243/) (2023)
- [The 2017 DAVIS Challenge on Video Object Segmentation](https://mlsanity.com/paper/1704.00675/) (2017)
- [iBOT: Image BERT Pre-Training with Online Tokenizer](https://mlsanity.com/paper/2111.07832/) (2021)
- [Weight Normalization: A Simple Reparameterization to Accelerate Training of Deep Neural Networks](https://mlsanity.com/paper/1602.07868/) (2016)
- [Cut and Learn for Unsupervised Object Detection and Instance Segmentation](https://mlsanity.com/paper/2301.11320/) (2023)
- [Learning Diverse and Discriminative Representations via the Principle of Maximal Coding Rate Reduction](https://mlsanity.com/paper/2006.08558/) (2020)
- [Stronger, Fewer, & Superior: Harnessing Vision Foundation Models for Domain Generalized Semantic Segmentation](https://mlsanity.com/paper/2312.04265/) (2023)
- [White-Box Transformers via Sparse Rate Reduction](https://mlsanity.com/paper/2306.01129/) (2023)
- [Neural Manifold Clustering and Embedding](https://mlsanity.com/paper/2201.10000/) (2022)
- [Learning Efficient Coding of Natural Images with Maximum Manifold Capacity Representations](https://mlsanity.com/paper/2303.03307/) (2023)
- [Connecting Joint-Embedding Predictive Architecture with Contrastive Self-supervised Learning](https://mlsanity.com/paper/2410.19560/) (2024)
- [Closed-Loop Data Transcription to an LDR via Minimaxing Rate Reduction](https://mlsanity.com/paper/2111.06636/) (2021)
- [Token Statistics Transformer: Linear-Time Attention via Variational Rate Reduction](https://mlsanity.com/paper/2412.17810/) (2024)

## Built on by (most cited first, 21 of 21)

- [SceneSplat: Gaussian Splatting-based Scene Understanding with Vision-Language Pretraining](https://mlsanity.com/paper/2503.18052/) (2025)
- [Learning Invariant Visual Representations for Planning with Joint-Embedding Predictive World Models](https://mlsanity.com/paper/2602.18639/) (2026)
- [BenchECG and xECG: a benchmark and baseline for ECG foundation models](https://mlsanity.com/paper/2509.10151/) (2025)
- [MuM: Multi-View Masked Image Modeling for 3D Vision](https://mlsanity.com/paper/2511.17309/) (2025)
- [Brain-Semantoks: Learning Semantic Tokens of Brain Dynamics with a Self-Distilled Foundation Model](https://mlsanity.com/paper/2512.11582/) (2025)
- [Kuramoto Oscillatory Phase Encoding: Neuro-inspired Synchronization for Improved Learning Efficiency](https://mlsanity.com/paper/2604.07904/) (2026)
- [Beyond Isotropy in JEPAs: Hamiltonian Geometry and Symplectic Prediction](https://mlsanity.com/paper/2605.20107/) (2026)
- [M3Ret: Unleashing Zero-shot Multimodal Medical Image Retrieval via Self-Supervision](https://mlsanity.com/paper/2509.01360/) (2025)
- [Image Hashing via Cross-View Code Alignment in the Age of Foundation Models](https://mlsanity.com/paper/2510.27584/) (2025)
- [Self-Supervised Learning as Discrete Communication](https://mlsanity.com/paper/2602.09764/) (2026)
- [Brewing Stronger Features: Dual-Teacher Distillation for Multispectral Earth Observation](https://mlsanity.com/paper/2602.19863/) (2026)
- [Hashing-Baseline: Rethinking Hashing in the Age of Pretrained Models](https://mlsanity.com/paper/2509.14427/) (2025)
- [Why Prototypes Collapse: Diagnosing and Preventing Partial Collapse in Prototypical Self-Supervised Learning](https://mlsanity.com/paper/2510.20108/) (2025)
- [Randomly Initialized Networks Can Learn from Peer-to-Peer Consensus](https://mlsanity.com/paper/2604.18390/) (2026)
- [Self-Supervised Learning of Plant Image Representations](https://mlsanity.com/paper/2604.27538/) (2026)
- [Unsupervised Semantic Segmentation Facilitates Model Understanding](https://mlsanity.com/paper/2605.29691/) (2026)
- [DINOcular: Self-Supervised Visuospatial Representations](https://mlsanity.com/paper/2608.27226/) (2026)
- [Unsupervised Point Cloud Registration via Training-Time Semantic Guidance](https://mlsanity.com/paper/2609.15228/) (2026)
- [On the Limits of Maximal Coding Rate Reduction for Out-of-Distribution Generalisation](https://mlsanity.com/paper/2609.21001/) (2026)
- [Two Global Crops Suffice: Locating Semantic Emergence in DINO-Style Self-Supervised Learning](https://mlsanity.com/paper/2609.28187/) (2026)
- [Masked Swingers: Harnessing Data Augmentation to Advance Autoencoders for Self-Supervised Learning](https://mlsanity.com/paper/2609.38278/) (2026)

## Abstract (arXiv)

DINO and DINOv2 are two model families being widely used to learn representations from unlabeled imagery data at large scales. Their learned representations often enable state-of-the-art performance for downstream tasks, such as image classification and segmentation. However, they employ many empirically motivated design choices and their training pipelines are highly complex and unstable -- many hyperparameters need to be carefully tuned to ensure that the representations do not collapse -- which poses considerable difficulty to improving them or adapting them to new domains. In this work, we posit that we can remove most such-motivated idiosyncrasies in the pre-training pipelines, and only need to add an explicit coding rate term in the loss function to avoid collapse of the representations. As a result, we obtain highly simplified variants of the DINO and DINOv2 which we call SimDINO and SimDINOv2, respectively. Remarkably, these simplified models are more robust to different design choices, such as network architecture and hyperparameters, and they learn even higher-quality representations, measured by performance on downstream tasks, offering a Pareto improvement over the corresponding DINO and DINOv2 models. This work highlights the potential of using simplifying design principles to improve the empirical practice of deep learning.