# Data Density Scaling Laws for Image Self-Distillation

arXiv: https://arxiv.org/abs/2610.19013 · HTML: https://mlsanity.com/paper/2610.19013/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Placed in General knowledge distillation methods, one of 12 directions in Knowledge distillation.
- Builds on 31 of its 52 references that are in the map.
- First posted on arXiv 13 Oct 2026.

## Placed in

- Knowledge distillation / [General knowledge distillation methods](https://mlsanity.com/distill/general-knowledge-distillation-methods/)

## Builds on (most cited first, 30 of 31)

- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [DINOv2: Learning Robust Visual Features without Supervision](https://mlsanity.com/paper/2304.07193/) (2023)
- [Masked Autoencoders Are Scalable Vision Learners](https://mlsanity.com/paper/2111.06377/) (2021)
- [Emerging Properties in Self-Supervised Vision Transformers](https://mlsanity.com/paper/2104.14294/) (2021)
- [Scaling Laws for Neural Language Models](https://mlsanity.com/paper/2001.08361/) (2020)
- [Training Compute-Optimal Large Language Models](https://mlsanity.com/paper/2203.15556/) (2022)
- [FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioning](https://mlsanity.com/paper/2307.08691/) (2023)
- [Scalability in Perception for Autonomous Driving: Waymo Open Dataset](https://mlsanity.com/paper/1912.04838/) (2019)
- [Describing Textures in the Wild](https://mlsanity.com/paper/1311.3618/) (2013)
- [Fine-Grained Visual Classification of Aircraft](https://mlsanity.com/paper/1306.5151/) (2013)
- [EuroSAT: A Novel Dataset and Deep Learning Benchmark for Land Use and Land Cover Classification](https://mlsanity.com/paper/1709.00029/) (2017)
- [BDD100K: A Diverse Driving Dataset for Heterogeneous Multitask Learning](https://mlsanity.com/paper/1805.04687/) (2018)
- [iBOT: Image BERT Pre-Training with Online Tokenizer](https://mlsanity.com/paper/2111.07832/) (2021)
- [Remote Sensing Image Scene Classification: Benchmark and State of the Art](https://mlsanity.com/paper/1703.00121/) (2017)
- [DataComp: In search of the next generation of multimodal datasets](https://mlsanity.com/paper/2304.14108/) (2023)
- [Rescaling Egocentric Vision](https://mlsanity.com/paper/2006.13256/) (2020)
- [Beyond neural scaling laws: beating power law scaling via data pruning](https://mlsanity.com/paper/2206.14486/) (2022)
- [Rotation Equivariant CNNs for Digital Pathology](https://mlsanity.com/paper/1806.03962/) (2018)
- [Scaling Data-Constrained Language Models](https://mlsanity.com/paper/2305.16264/) (2023)
- [AID: A Benchmark Dataset for Performance Evaluation of Aerial Scene Classification](https://mlsanity.com/paper/1608.05167/) (2016)
- [The Vendi Score: A Diversity Evaluation Metric for Machine Learning](https://mlsanity.com/paper/2210.02410/) (2022)
- [SemDeDup: Data-efficient learning at web-scale through semantic deduplication](https://mlsanity.com/paper/2303.09540/) (2023)
- [Challenges in Representation Learning: A report on three machine learning contests](https://mlsanity.com/paper/1307.0414/) (2013)
- [CORe50: a New Dataset and Benchmark for Continuous Object Recognition](https://mlsanity.com/paper/1705.03550/) (2017)
- [SatlasPretrain: A Large-Scale Dataset for Remote Sensing Image Understanding](https://mlsanity.com/paper/2211.15660/) (2022)
- [PatternNet: A Benchmark Dataset for Performance Evaluation of Remote Sensing Image Retrieval](https://mlsanity.com/paper/1706.03424/) (2017)
- [Scaling Laws for Data Filtering -- Data Curation cannot be Compute Agnostic](https://mlsanity.com/paper/2404.07177/) (2024)
- [EgoObjects: A Large-Scale Egocentric Dataset for Fine-Grained Object Understanding](https://mlsanity.com/paper/2309.08816/) (2023)
- [Most Influential Subset Selection: Challenges, Promises, and Beyond](https://mlsanity.com/paper/2409.18153/) (2024)
- [Cram Less to Fit More: Training Data Pruning Improves Memorization of Facts](https://mlsanity.com/paper/2604.08519/) (2026)

## Abstract (arXiv)

Current scaling law formulations suggest that increasing the number of unique samples in a dataset improves model performance, while repeated exposure to the same samples provides diminishing value. In this work, we examine this assumption for image self-distillation models. Through systematic experiments across diverse datasets, namely Maxar, SATLAS, Waymo, nuPlan, Ego4D, ImageNet, and YFCC15M, we demonstrate that the effects of sample repetition on downstream performance are non-trivial and dataset-dependent. We identify settings where repeatedly training on a smaller random subset can outperform training on a larger set of unique samples. This finding highlights a limitation of existing scaling laws. To address this gap, we propose a modified version of a Chinchilla-style scaling law that explicitly accounts for two factors: the number of unique samples and repetition per sample. This formulation provides a framework for understanding data-compute trade-offs and improves prediction in some regimes where repetition plays a central role, although its benefits are not consistent across datasets. Finally, we examine how subset selection and augmentation affect downstream performance.