mlsanityData to 7 Oct 2026

Scalable Pre-training of Large Autoregressive Image Models

arXiv 2401.08541 ↗First posted 16 Jan 2024cs.CVcited by 115

Not on the map. Why?

Abstract (arXiv)

This paper introduces AIM, a collection of vision models pre-trained with an autoregressive objective. These models are inspired by their textual counterparts, i.e., Large Language Models (LLMs), and exhibit similar scaling properties. Specifically, we highlight two key findings: (1) the performance of the visual features scale with both the model capacity and the quantity of data, (2) the value of the objective function correlates with the performance of the model on downstream tasks. We illustrate the practical implication of these findings by pre-training a 7 billion parameter AIM on 2 billion images, that achieves 84.0% on ImageNet-1k with a frozen trunk. Interestingly, even at this scale, we observe no sign of saturation in performance, suggesting that AIM potentially represents a new frontier for training large-scale vision models. The pre-training of AIM is similar to the pre-training of LLMs, and does not require any image-specific strategy to stabilize the training at scale.

Key facts. Cited by 115 papers in the map (data through 7 Oct 2026). Builds on 62 of its 84 references that are in the map. First posted on arXiv 16 Jan 2024.

Builds on

62 of 84 references.

References · 62

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2020Language Models are Few-Shot Learnerscited by 32,199
  4. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  5. 2022Training language models to follow instructions with human feedbackcited by 15,712
  6. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  7. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  8. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  9. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 11,979
  10. 2018Representation Learning with Contrastive Predictive Codingcited by 7,662
  11. 2019Momentum Contrast for Unsupervised Visual Representation Learningcited by 7,482
  12. 2021Masked Autoencoders Are Scalable Vision Learnerscited by 7,339
38 more
  1. 2017Mask R-CNNcited by 7,259
  2. 2021Emerging Properties in Self-Supervised Vision Transformerscited by 6,893
  3. 2017mixup: Beyond Empirical Risk Minimizationcited by 5,155
  4. 2016SGDR: Stochastic Gradient Descent with Warm Restartscited by 4,993
  5. 2020Bootstrap your own latent: A new approach to self-supervised Learningcited by 4,712
  6. 2020Training data-efficient image transformers & distillation through attentioncited by 4,560
  7. 2021Zero-Shot Text-to-Image Generationcited by 4,276
  8. 2022Training Compute-Optimal Large Language Modelscited by 2,927
  9. 2020Unsupervised Learning of Visual Features by Contrasting Cluster Assignmentscited by 2,845
  10. 2018Large Scale GAN Training for High Fidelity Natural Image Synthesiscited by 2,609
  11. 2021BEiT: BERT Pre-Training of Image Transformerscited by 2,360
  12. 2013Describing Textures in the Wildcited by 2,110
  13. 2021Barlow Twins: Self-Supervised Learning via Redundancy Reductioncited by 1,818
  14. 2016WaveNet: A Generative Model for Raw Audiocited by 1,710
  15. 2021An Empirical Study of Training Self-Supervised Vision Transformerscited by 1,610
  16. 2018Unsupervised Representation Learning by Predicting Image Rotationscited by 1,586
  17. 2017EuroSAT: A Novel Dataset and Deep Learning Benchmark for Land Use and Land Cover Classificationcited by 1,488
  18. 2016Context Encoders: Feature Learning by Inpaintingcited by 1,424
  19. 2016Unsupervised Learning of Visual Representations by Solving Jigsaw Puzzlescited by 1,379
  20. 2018Moment Matching for Multi-Source Domain Adaptationcited by 1,327
  21. 2022CoCa: Contrastive Captioners are Image-Text Foundation Modelscited by 1,265
  22. 2016Colorful Image Colorizationcited by 1,113
  23. 2015Unsupervised Visual Representation Learning by Context Predictioncited by 1,062
  24. 2018Deep Clustering for Unsupervised Learning of Visual Featurescited by 1,020
  25. 2023Self-Supervised Learning from Images with a Joint-Embedding Predictive Architecturecited by 982
  26. 2016Deep Networks with Stochastic Depthcited by 973
  27. 2021iBOT: Image BERT Pre-Training with Online Tokenizercited by 891
  28. 2016Conditional Image Generation with PixelCNN Decoderscited by 859
  29. 2016Pixel Recurrent Neural Networkscited by 814
  30. 2019Self-Supervised Learning of Pretext-Invariant Representationscited by 806
  31. 2018Image Transformercited by 770
  32. 2021Going deeper with Image Transformerscited by 759
  33. 2023Scaling Vision Transformers to 22 Billion Parameterscited by 690
  34. 2023DataComp: In search of the next generation of multimodal datasetscited by 635
  35. 2018Rotation Equivariant CNNs for Digital Pathologycited by 420
  36. 2017PixelCNN++: Improving the PixelCNN with Discretized Logistic Mixture Likelihood and Other Modificationscited by 415
  37. 2017Functional Map of the Worldcited by 338
  38. 2014Discriminative Unsupervised Feature Learning with Exemplar Convolutional Neural Networkscited by 299

50 most cited of 62 All 62 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 54; 2025: 43; 2026: 18.

115 papers.

Sort:

By area:

  1. 2025SmolVLA: A Vision-Language-Action Model for Affordable and Efficient Roboticscited by 507
  2. 2025Perception Encoder: The best visual embeddings are not at the output of the networkcited by 315
  3. 2024An Image is Worth 32 Tokens for Reconstruction and Generationcited by 295
  4. 2025Layer by Layer: Uncovering Hidden Representations in Language Modelscited by 293
  5. 2024MM1: Methods, Analysis & Insights from Multimodal LLM Pre-trainingcited by 230
  6. 2024Sapiens: Foundation for Human Vision Modelscited by 229
  7. 2024Fluid: Scaling Autoregressive Text-to-image Generative Models with Continuous Tokenscited by 147
  8. 2024Scaling Proprioceptive-Visual Learning with Heterogeneous Pre-trained Transformerscited by 145
  9. 2024Randomized Autoregressive Visual Generationcited by 125
  10. 2024Multimodal Autoregressive Pre-training of Large Vision Encoderscited by 108
  11. 2024Unveiling Encoder-Free Vision-Language Modelscited by 90
  12. 2024Learning and Leveraging World Models in Visual Representation Learningcited by 71
38 more
  1. 2024When Do We Not Need Larger Vision Models?cited by 65
  2. 20244M-21: An Any-to-Any Vision Model for Tens of Tasks and Modalitiescited by 52
  3. 2025TokLIP: Marry Visual Tokens to CLIP for Multimodal Comprehension and Generationcited by 50
  4. 2025Scaling Laws for Optimal Data Mixturescited by 49
  5. 2024CXPMRG-Bench: Pre-training and Benchmarking for X-ray Medical Report Generation on CheXpert Plus Datasetcited by 33
  6. 2025Interpreting the linear structure of vision-language model embedding spacescited by 32
  7. 2024How Far Are We From AGI: Are LLMs All We Need?cited by 31
  8. 2024Assessing and Learning Alignment of Unimodal Vision and Language Modelscited by 31
  9. 2024Scaling 4D Representationscited by 30
  10. 2025The Scalability of Simplicity: Empirical Analysis of Vision-Language Learning with a Single Transformercited by 28
  11. 2024Next Token Prediction Towards Multimodal Intelligence: A Comprehensive Surveycited by 27
  12. 2024DiffCut: Catalyzing Zero-Shot Semantic Segmentation with Diffusion Features and Recursive Normalized Cutcited by 26
  13. 2025Scaling Vision Pre-Training to 4K Resolutioncited by 26
  14. 2025Can Masked Autoencoders Also Listen to Birds?cited by 26
  15. 2024Stabilize the Latent Space for Image Autoregressive Modeling: A Unified Perspectivecited by 22
  16. 2024Autoregressive Pretraining with Mamba in Visioncited by 21
  17. 2025An Empirical Study of Autoregressive Pre-training from Videoscited by 21
  18. 2025Next-Embedding Prediction Makes Strong Vision Learnerscited by 21
  19. 2024DepthCues: Evaluating Monocular Depth Perception in Large Vision Modelscited by 18
  20. 2024MiM: Mask in Mask Self-Supervised Pre-Training for 3D Medical Image Analysiscited by 17
  21. 2025Cluster and Predict Latent Patches for Improved Masked Image Modelingcited by 17
  22. 2024You Don't Need Domain-Specific Data Augmentations When Scaling Self-Supervised Learningcited by 16
  23. 2024Data-efficient Large Vision Models through Sequential Autoregressioncited by 14
  24. 2024When Worse is Better: Navigating the compression-generation tradeoff in visual tokenizationcited by 14
  25. 2024Locality Alignment Improves Vision-Language Modelscited by 13
  26. 2025Region-based Cluster Discrimination for Visual Representation Learningcited by 13
  27. 2024Moving Off-the-Grid: Scene-Grounded Video Representationscited by 11
  28. 2025RoMA: Scaling up Mamba-based Foundation Models for Remote Sensingcited by 11
  29. 2025Attention, Please! Revisiting Attentive Probing Through the Lens of Efficiencycited by 11
  30. 2025Hita: Holistic Tokenizer for Autoregressive Image Generationcited by 11
  31. 2025Foundation Models for Bioacoustics -- a Comparative Reviewcited by 11
  32. 2025Rethinking JEPA: Compute-Efficient Video SSL with Frozen Teacherscited by 11
  33. 2024Autoregressive Pre-Training on Pixels and Textscited by 10
  34. 2024Towards Unifying Understanding and Generation in the Era of Vision Foundation Models: A Survey from the Autoregression Perspectivecited by 10
  35. 2025Learning Streaming Video Representation via Multitask Trainingcited by 10
  36. 2026OneVision-Encoder: Codec-Aligned Sparsity as a Foundational Principle for Multimodal Intelligencecited by 10
  37. 2024Denoising Autoregressive Representation Learningcited by 9
  38. 2024AstroPT: Scaling Large Observation Models for Astronomycited by 9

50 most cited of 115 All 115 (JSON)