mlsanityData to 14 Oct 2026

An Empirical Study of Autoregressive Pre-training from Videos

arXiv 2501.05453 ↗First posted 9 Jan 2025cs.CV cs.AIcited by 21

Not on the map. Why?

Abstract (arXiv)

We empirically study autoregressive pre-training from videos. To perform our study, we construct a series of autoregressive video models, called Toto. We treat videos as sequences of visual tokens and train transformer models to autoregressively predict future tokens. Our models are pre-trained on a diverse dataset of videos and images comprising over 1 trillion visual tokens. We explore different architectural, training, and inference design choices. We evaluate the learned visual representations on a range of downstream tasks including image recognition, video classification, object tracking, and robotics. Our results demonstrate that, despite minimal inductive biases, autoregressive pre-training leads to competitive performance across all benchmarks. Finally, we find that scaling our video models results in similar scaling curves to those seen in language models, albeit with a different rate. More details at https://brjathu.github.io/toto/

Key facts. Cited by 21 papers in the map (data through 14 Oct 2026). Builds on 54 of its 66 references that are in the map. First posted on arXiv 9 Jan 2025.

Builds on

54 of 66 references.

References · 54

  1. 2017Attention Is All You Needcited by 60,361
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  3. 2020Language Models are Few-Shot Learnerscited by 32,419
  4. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,863
  5. 2014Very Deep Convolutional Networks for Large-Scale Image Recognitioncited by 15,364
  6. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  7. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 12,049
  8. 2014ImageNet Large Scale Visual Recognition Challengecited by 8,498
  9. 2023DINOv2: Learning Robust Visual Features without Supervisioncited by 7,974
  10. 2019Momentum Contrast for Unsupervised Visual Representation Learningcited by 7,513
  11. 2021Masked Autoencoders Are Scalable Vision Learnerscited by 7,403
  12. 2021Emerging Properties in Self-Supervised Vision Transformerscited by 6,968
38 more
  1. 2020Bootstrap your own latent: A new approach to self-supervised Learningcited by 4,750
  2. 2023Mamba: Linear-Time Sequence Modeling with Selective State Spacescited by 4,718
  3. 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,703
  4. 2021Zero-Shot Text-to-Image Generationcited by 4,288
  5. 2020Taming Transformers for High-Resolution Image Synthesiscited by 3,253
  6. 2020Unsupervised Learning of Visual Features by Contrasting Cluster Assignmentscited by 2,857
  7. 2016Perceptual Losses for Real-Time Style Transfer and Super-Resolutioncited by 2,694
  8. 2021BEiT: BERT Pre-Training of Image Transformerscited by 2,375
  9. 2017The Kinetics Human Action Video Datasetcited by 1,909
  10. 2018SlowFast Networks for Video Recognitioncited by 1,867
  11. 2018Unsupervised Feature Learning via Non-Parametric Instance-level Discriminationcited by 1,845
  12. 2020GLU Variants Improve Transformercited by 1,804
  13. 2021Ego4D: Around the World in 3,000 Hours of Egocentric Videocited by 1,556
  14. 2019Root Mean Square Layer Normalizationcited by 1,480
  15. 2022Reproducible scaling laws for contrastive language-image learningcited by 1,173
  16. 2023Self-Supervised Learning from Images with a Joint-Embedding Predictive Architecturecited by 1,039
  17. 2022In-context Learning and Induction Headscited by 964
  18. 2017The 2017 DAVIS Challenge on Video Object Segmentationcited by 957
  19. 2019HowTo100M: Learning a Text-Video Embedding by Watching Hundred Million Narrated Video Clipscited by 950
  20. 2016Conditional Image Generation with PixelCNN Decoderscited by 862
  21. 2016Pixel Recurrent Neural Networkscited by 816
  22. 2018Image Transformercited by 772
  23. 2013Network In Networkcited by 605
  24. 2023VideoMAE V2: Scaling Video Masked Autoencoders with Dual Maskingcited by 528
  25. 2020Scaling Laws for Autoregressive Generative Modelingcited by 522
  26. 2022InternVideo: General Video Foundation Models via Generative and Discriminative Learningcited by 447
  27. 2022Masked Autoencoders As Spatiotemporal Learnerscited by 429
  28. 2015Autoencoding beyond pixels using a learned similarity metriccited by 407
  29. 2019A Short Note on the Kinetics-700 Human Action Datasetcited by 310
  30. 2023Hiera: A Hierarchical Vision Transformer without the Bells-and-Whistlescited by 290
  31. 2020Understanding Human Hands in Contact at Internet Scalecited by 281
  32. 2022Masked Visual Pre-training for Motor Controlcited by 248
  33. 2022Real-World Robot Learning with Masked Visual Pre-trainingcited by 245
  34. 2020Space-Time Correspondence as a Contrastive Random Walkcited by 239
  35. 2019Scaling Autoregressive Video Modelscited by 190
  36. 2019CATER: A diagnostic dataset for Compositional Actions and TEmporal Reasoningcited by 137
  37. 2024Scalable Pre-training of Large Autoregressive Image Modelscited by 115
  38. 2014Video (language) modeling: a baseline for generative models of natural videoscited by 92

50 most cited of 54 All 54 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 13; 2026: 8.

21 papers.

Sort:

By area:

  1. 2025V-JEPA 2: Self-Supervised Video Models Enable Understanding, Prediction and Planningcited by 809
  2. 2025SmolVLA: A Vision-Language-Action Model for Affordable and Efficient Roboticscited by 530
  3. 2025Perception Encoder: The best visual embeddings are not at the output of the networkcited by 326
  4. 2026V-JEPA 2.1: Unlocking Dense Features in Video Self-Supervised Learningcited by 117
  5. 2025Scaling Laws for Optimal Data Mixturescited by 52
  6. 2025Scaling Laws for Native Multimodal Modelscited by 50
  7. 2025VaViM and VaVAM: Autonomous Driving through Video Generative Modelingcited by 35
  8. 2026Past- and Future-Informed KV Cache Policy with Salience Estimation in Autoregressive Video Diffusioncited by 16
  9. 2025Chirality in Action: Time-Aware Video Representation Learning by Latent Straighteningcited by 11
  10. 2025Learning Streaming Video Representation via Multitask Trainingcited by 10
  11. 2026Attend Before Attention: Efficient and Scalable Video Understanding via Autoregressive Gazingcited by 9
  12. 2026VideoFlexTok: Flexible-Length Coarse-to-Fine Video Tokenizationcited by 6
9 more
  1. 2025Frozen Forecasting: A Unified Evaluationcited by 4
  2. 2026Uncovering the Latent Potential of Deep Intermediate Representationscited by 3
  3. 2025Poly-Autoregressive Prediction for Modeling Interactionscited by 1
  4. 2025AutoCas: Autoregressive Cascade Predictor in Social Networks via Large Language Modelscited by 1
  5. 2025Learning from Next-Frame Prediction: Autoregressive Video Modeling Encodes Effective Representationscited by 1
  6. 2025Tracking by Predicting 3-D Gaussians Over Timecited by 0
  7. 2026Separators in Enhancing Autoregressive Pretraining for Vision Mambacited by 0
  8. 2026What Cohort INRs Encode and Where to Freeze Themcited by 0
  9. 2026TT-VidT: Decoupling the Temporal Axis for Efficient Motion-Centric Video Pretrainingcited by 0