# Is ImageNet worth 1 video? Learning strong image encoders from 1 long unlabelled video

arXiv: https://arxiv.org/abs/2310.08584 · HTML: https://mlsanity.com/paper/2310.08584/

## Key facts

- Cited by 40 papers in the map (data through 14 Oct 2026).
- Builds on 41 of its 67 references that are in the map.
- First posted on arXiv 12 Oct 2023.

## Builds on (most cited first, 30 of 41)

- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [A Simple Framework for Contrastive Learning of Visual Representations](https://mlsanity.com/paper/2002.05709/) (2020)
- [DINOv2: Learning Robust Visual Features without Supervision](https://mlsanity.com/paper/2304.07193/) (2023)
- [Emerging Properties in Self-Supervised Vision Transformers](https://mlsanity.com/paper/2104.14294/) (2021)
- [Unsupervised Learning of Visual Features by Contrasting Cluster Assignments](https://mlsanity.com/paper/2006.09882/) (2020)
- [Sinkhorn Distances: Lightspeed Computation of Optimal Transportation Distances](https://mlsanity.com/paper/1306.0895/) (2013)
- [The Kinetics Human Action Video Dataset](https://mlsanity.com/paper/1705.06950/) (2017)
- [Ego4D: Around the World in 3,000 Hours of Egocentric Video](https://mlsanity.com/paper/2110.07058/) (2021)
- [Unified Perceptual Parsing for Scene Understanding](https://mlsanity.com/paper/1807.10221/) (2018)
- [Frozen in Time: A Joint Video and Image Encoder for End-to-End Retrieval](https://mlsanity.com/paper/2104.00650/) (2021)
- [VICReg: Variance-Invariance-Covariance Regularization for Self-Supervised Learning](https://mlsanity.com/paper/2105.04906/) (2021)
- [The 2017 DAVIS Challenge on Video Object Segmentation](https://mlsanity.com/paper/1704.00675/) (2017)
- [HowTo100M: Learning a Text-Video Embedding by Watching Hundred Million Narrated Video Clips](https://mlsanity.com/paper/1906.03327/) (2019)
- [iBOT: Image BERT Pre-Training with Online Tokenizer](https://mlsanity.com/paper/2111.07832/) (2021)
- [Detecting Twenty-thousand Classes using Image-level Supervision](https://mlsanity.com/paper/2201.02605/) (2022)
- [Rescaling Egocentric Vision](https://mlsanity.com/paper/2006.13256/) (2020)
- [Self-labelling via simultaneous clustering and representation learning](https://mlsanity.com/paper/1911.05371/) (2019)
- [GOT-10k: A Large High-Diversity Benchmark for Generic Object Tracking in the Wild](https://mlsanity.com/paper/1810.11981/) (2018)
- [Time-Contrastive Networks: Self-Supervised Learning from Video](https://mlsanity.com/paper/1704.06888/) (2017)
- [AVA: A Video Dataset of Spatio-temporally Localized Atomic Visual Actions](https://mlsanity.com/paper/1705.08421/) (2017)
- [Cascade R-CNN: High Quality Object Detection and Instance Segmentation](https://mlsanity.com/paper/1906.09756/) (2019)
- [Assembly101: A Large-Scale Multi-View Video Dataset for Understanding Procedural Activities](https://mlsanity.com/paper/2203.14712/) (2022)
- [Learning Correspondence from the Cycle-Consistency of Time](https://mlsanity.com/paper/1903.07593/) (2019)
- [Localizing Objects with Self-Supervised Transformers and no Labels](https://mlsanity.com/paper/2109.14279/) (2021)
- [Learning Features by Watching Objects Move](https://mlsanity.com/paper/1612.06370/) (2016)
- [What to Hide from Your Students: Attention-Guided Masked Image Modeling](https://mlsanity.com/paper/2203.12719/) (2022)
- [Solo-learn: A Library of Self-supervised Methods for Visual Representation Learning](https://mlsanity.com/paper/2108.01775/) (2021)
- [Learning to See by Moving](https://mlsanity.com/paper/1505.01596/) (2015)
- [Joint-task Self-supervised Learning for Temporal Correspondence](https://mlsanity.com/paper/1909.11895/) (2019)

## Built on by (most cited first, 30 of 40)

- [Audio Flamingo 2: An Audio-Language Model with Long-Audio Understanding and Expert Reasoning Abilities](https://mlsanity.com/paper/2503.03983/) (2025)
- [AnyCam: Learning to Recover Camera Poses and Intrinsics from Casual Videos](https://mlsanity.com/paper/2503.23282/) (2025)
- [Scaling Backwards: Minimal Synthetic Pre-training?](https://mlsanity.com/paper/2408.00677/) (2024)
- [Frozen-DETR: Enhancing DETR with Image Understanding from Frozen Foundation Models](https://mlsanity.com/paper/2410.19635/) (2024)
- [LesionLocator: Zero-Shot Universal Tumor Segmentation and Tracking in 3D Whole-Body Imaging](https://mlsanity.com/paper/2502.20985/) (2025)
- [Learning from One Continuous Video Stream](https://mlsanity.com/paper/2312.00598/) (2023)
- [PooDLe: Pooled and dense self-supervised learning from naturalistic videos](https://mlsanity.com/paper/2408.11208/) (2024)
- [Chirality in Action: Time-Aware Video Representation Learning by Latent Straightening](https://mlsanity.com/paper/2509.08502/) (2025)
- [Direct Motion Models for Assessing Generated Videos](https://mlsanity.com/paper/2505.00209/) (2025)
- [DiLA: Disentangled Latent Action World Models](https://mlsanity.com/paper/2605.15725/) (2026)
- [Learning from Streaming Video with Orthogonal Gradients](https://mlsanity.com/paper/2504.01961/) (2025)
- [EgoAgent: A Joint Predictive Agent Model in Egocentric Worlds](https://mlsanity.com/paper/2502.05857/) (2025)
- [Beyond [cls]: Exploring the true potential of Masked Image Modeling representations](https://mlsanity.com/paper/2412.03215/) (2024)
- [Memory Storyboard: Leveraging Temporal Segmentation for Streaming Self-Supervised Learning from Egocentric Videos](https://mlsanity.com/paper/2501.12254/) (2025)
- [Perception Test 2024: Challenge Summary and a Novel Hour-Long VideoQA Benchmark](https://mlsanity.com/paper/2411.19941/) (2024)
- [TRecViT: A Recurrent Video Transformer](https://mlsanity.com/paper/2412.14294/) (2024)
- [MoSiC: Optimal-Transport Motion Trajectory for Dense Self-Supervised Learning](https://mlsanity.com/paper/2506.08694/) (2025)
- [Universal pre-training by iterated random computation](https://mlsanity.com/paper/2506.20057/) (2025)
- [Language-based Audio Moment Retrieval](https://mlsanity.com/paper/2409.15672/) (2024)
- [LayerLock: Non-collapsing Representation Learning with Progressive Freezing](https://mlsanity.com/paper/2509.10156/) (2025)
- [CASTELLA: Long Audio Dataset with Captions and Temporal Boundaries](https://mlsanity.com/paper/2511.15131/) (2025)
- [LeVJEPA: Efficient & Scalable Video Pretraining without the Heuristics](https://mlsanity.com/paper/2608.27395/) (2026)
- [Midway Network: Learning Representations for Recognition and Motion from Latent Dynamics](https://mlsanity.com/paper/2510.05558/) (2025)
- [Unique Lives, Shared World: Learning from Single-Life Videos](https://mlsanity.com/paper/2512.04085/) (2025)
- [Human Gaze Boosts Object-Centered Representation Learning](https://mlsanity.com/paper/2501.02966/) (2025)
- [Consistent View Alignment Improves Foundation Models for 3D Medical Image Segmentation](https://mlsanity.com/paper/2509.13846/) (2025)
- [SpaRRTa: A Synthetic Benchmark for Evaluating Spatial Intelligence in Visual Foundation Models](https://mlsanity.com/paper/2601.11729/) (2026)
- [FALCONEye: Finding Answers and Localizing Content in ONE-hour-long videos with multi-modal LLMs](https://mlsanity.com/paper/2503.19850/) (2025)
- [Visual Pre-Training on Unlabeled Images using Reinforcement Learning](https://mlsanity.com/paper/2506.11967/) (2025)
- [Video Self-Distillation for Single-Image Encoders: A Step Toward Physically Plausible Perception](https://mlsanity.com/paper/2507.19272/) (2025)

## Abstract (arXiv)

Self-supervised learning has unlocked the potential of scaling up pretraining to billions of images, since annotation is unnecessary. But are we making the best use of data? How more economical can we be? In this work, we attempt to answer this question by making two contributions. First, we investigate first-person videos and introduce a "Walking Tours" dataset. These videos are high-resolution, hours-long, captured in a single uninterrupted take, depicting a large number of objects and actions with natural scene transitions. They are unlabeled and uncurated, thus realistic for self-supervision and comparable with human learning. Second, we introduce a novel self-supervised image pretraining method tailored for learning from continuous videos. Existing methods typically adapt image-based pretraining approaches to incorporate more frames. Instead, we advocate a "tracking to learn to recognize" approach. Our method called DoRA, leads to attention maps that Discover and tRAck objects over time in an end-to-end manner, using transformer cross-attention. We derive multiple views from the tracks and use them in a classical self-supervised distillation loss. Using our novel approach, a single Walking Tours video remarkably becomes a strong competitor to ImageNet for several image and video downstream tasks.