mlsanityData to 7 Oct 2026

Sequential Modeling Enables Scalable Learning for Large Vision Models

arXiv 2312.00785 ↗First posted 1 Dec 2023cs.CVcited by 212

Not on the map. Why?

Abstract (arXiv)

We introduce a novel sequential modeling approach which enables learning a Large Vision Model (LVM) without making use of any linguistic data. To do this, we define a common format, "visual sentences", in which we can represent raw images and videos as well as annotated data sources such as semantic segmentations and depth reconstructions without needing any meta-knowledge beyond the pixels. Once this wide variety of visual data (comprising 420 billion tokens) is represented as sequences, the model can be trained to minimize a cross-entropy loss for next token prediction. By training across various scales of model architecture and data diversity, we provide empirical evidence that our models scale effectively. Many different vision tasks can be solved by designing suitable visual prompts at test time.

Key facts. Cited by 212 papers in the map (data through 7 Oct 2026). Builds on 59 of its 99 references that are in the map. First posted on arXiv 1 Dec 2023.

Builds on

59 of 99 references.

References · 59

  1. 2017Attention Is All You Needcited by 59,932
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  4. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  5. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  6. 2023Visual Instruction Tuningcited by 7,792
  7. 2021Masked Autoencoders Are Scalable Vision Learnerscited by 7,339
  8. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
  9. 2020Bootstrap your own latent: A new approach to self-supervised Learningcited by 4,712
  10. 2017Neural Discrete Representation Learningcited by 4,650
  11. 2016The Cityscapes Dataset for Semantic Urban Scene Understandingcited by 4,427
  12. 2022LAION-5B: An open large-scale dataset for training next generation image-text modelscited by 4,131
38 more
  1. 2020Taming Transformers for High-Resolution Image Synthesiscited by 3,229
  2. 2020Exploring Simple Siamese Representation Learningcited by 2,853
  3. 2022InstructPix2Pix: Learning to Follow Image Editing Instructionscited by 2,591
  4. 2021BEiT: BERT Pre-Training of Image Transformerscited by 2,360
  5. 2021Masked-attention Mask Transformer for Universal Image Segmentationcited by 2,338
  6. 2021Vision Transformers for Dense Predictioncited by 1,789
  7. 2021LAION-400M: Open Dataset of CLIP-Filtered 400 Million Image-Text Pairscited by 1,544
  8. 2021Ego4D: Around the World in 3,000 Hours of Egocentric Videocited by 1,530
  9. 2022Objaverse: A Universe of Annotated 3D Objectscited by 1,522
  10. 2016Context Encoders: Feature Learning by Inpaintingcited by 1,424
  11. 2016Unsupervised Learning of Visual Representations by Solving Jigsaw Puzzlescited by 1,379
  12. 2019MMDetection: Open MMLab Detection Toolbox and Benchmarkcited by 1,379
  13. 2021SimMIM: A Simple Framework for Masked Image Modelingcited by 1,147
  14. 2016Colorful Image Colorizationcited by 1,113
  15. 2017The "something something" video database for learning and evaluating visual common sensecited by 1,092
  16. 2022MaskGIT: Masked Generative Image Transformercited by 1,082
  17. 2016Semantic Understanding of Scenes through the ADE20K Datasetcited by 1,063
  18. 2015Unsupervised Visual Representation Learning by Context Predictioncited by 1,062
  19. 2019Learning Robust Global Representations by Penalizing Local Predictive Powercited by 1,010
  20. 2016Stacked Hourglass Networks for Human Pose Estimationcited by 913
  21. 2016Conditional Image Generation with PixelCNN Decoderscited by 859
  22. 2016Pixel Recurrent Neural Networkscited by 814
  23. 2018Multi-Task Learning as Multi-Objective Optimizationcited by 809
  24. 2018Deep Retinex Decomposition for Low-Light Enhancementcited by 720
  25. 2021Vector-quantized Image Modeling with Improved VQGANcited by 656
  26. 2021Common Objects in 3D: Large-Scale Learning and Evaluation of Real-life 3D Category Reconstructioncited by 646
  27. 2018Taskonomy: Disentangling Task Transfer Learningcited by 631
  28. 2021Perceiver IO: A General Architecture for Structured Inputs & Outputscited by 616
  29. 2016Hollywood in Homes: Crowdsourcing Data Collection for Activity Understandingcited by 535
  30. 2018YouTube-VOS: A Large-Scale Video Object Segmentation Benchmarkcited by 381
  31. 2022Images Speak in Images: A Generalist Painter for In-Context Visual Learningcited by 317
  32. 2019A Short Note on the Kinetics-700 Human Action Datasetcited by 308
  33. 2018Density-aware Single Image De-raining using a Multi-stream Dense Networkcited by 248
  34. 2022Visual Prompting via Image Inpaintingcited by 248
  35. 2018Moments in Time Dataset: one million videos for event understandingcited by 215
  36. 2017Unite the People: Closing the Loop Between 3D and 2D Human Representationscited by 205
  37. 2016UberNet: Training a `Universal' Convolutional Neural Network for Low-, Mid-, and High-Level Vision using Diverse Datasets and Limited Memorycited by 200
  38. 2017Multi-task Self-Supervised Visual Learningcited by 200

50 most cited of 59 All 59 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 6; 2024: 104; 2025: 76; 2026: 26.

212 papers.

Sort:

By area:

  1. 2024Visual Autoregressive Modeling: Scalable Image Generation via Next-Scale Predictioncited by 1,005
  2. 2024Autoregressive Model Beats Diffusion: Llama for Scalable Image Generationcited by 760
  3. 2024Parameter-Efficient Fine-Tuning for Large Models: A Comprehensive Surveycited by 641
  4. 2024An Image is Worth 32 Tokens for Reconstruction and Generationcited by 295
  5. 2024Sapiens: Foundation for Human Vision Modelscited by 229
  6. 2024Fluid: Scaling Autoregressive Text-to-image Generative Models with Continuous Tokenscited by 147
  7. 2024ShapeLLM: Universal 3D Object Understanding for Embodied Interactioncited by 133
  8. 2024Randomized Autoregressive Visual Generationcited by 125
  9. 2025Long-Context Autoregressive Video Modeling with Next-Frame Predictioncited by 119
  10. 2024Scalable Pre-training of Large Autoregressive Image Modelscited by 115
  11. 2024ANOLE: An Open, Autoregressive, Native Large Multimodal Models for Interleaved Image-Text Generationcited by 112
  12. 2024Is Sora a World Simulator? A Comprehensive Survey on General World Models and Beyondcited by 111
38 more
  1. 2025MineWorld: a Real-Time and Open-Source Interactive World Model on Minecraftcited by 93
  2. 2024UniST: A Prompt-Empowered Universal Model for Urban Spatio-Temporal Predictioncited by 86
  3. 2024Video as the New Language for Real-World Decision Makingcited by 86
  4. 2025Visual Planning: Let's Think Only with Imagescited by 85
  5. 2024Position: Graph Foundation Models are Already Herecited by 84
  6. 2024GFT: Graph Foundation Model with Transferable Tree Vocabularycited by 82
  7. 2024Reloc3r: Large-Scale Training of Relative Camera Pose Regression for Generalizable, Fast, and Accurate Visual Localizationcited by 81
  8. 2024What Matters When Repurposing Diffusion Models for General Dense Perception Tasks?cited by 74
  9. 2025RayZer: A Self-supervised Large View Synthesis Modelcited by 74
  10. 2024OMG-Seg: Is One Model Good Enough For All Segmentation?cited by 73
  11. 2023Free3D: Consistent Novel View Synthesis without 3D Representationcited by 72
  12. 2024A Survey on Visual Anomaly Detection: Challenge, Approach, and Prospectcited by 69
  13. 2024BRAVE: Broadening the visual encoding of vision-language modelscited by 68
  14. 2024Scaling the Codebook Size of VQGAN to 100,000 with a Utilization Rate of 99%cited by 68
  15. 2024When Do We Not Need Larger Vision Models?cited by 65
  16. 2024In-Context Imitation Learning via Next-Token Predictioncited by 65
  17. 2024Moto: Latent Motion Token as the Bridging Language for Learning Robot Manipulation from Videoscited by 65
  18. 2024AbdomenAtlas: A Large-Scale, Detailed-Annotated, & Multi-Center Dataset for Efficient Transfer Learning and Open Algorithmic Benchmarkingcited by 64
  19. 2025Scaling Language-Free Visual Representation Learningcited by 62
  20. 2024ControlVAR: Exploring Controllable Visual Autoregressive Modelingcited by 58
  21. 2024GOFA: A Generative One-For-All Model for Joint Graph Language Modelingcited by 56
  22. 2024Knowledge Graphs Meet Multi-Modal Learning: A Comprehensive Surveycited by 50
  23. 2024Vitron: A Unified Pixel-level Vision LLM for Understanding, Generating, Segmenting, Editingcited by 49
  24. 2024UnlearnCanvas: Stylized Image Dataset for Enhanced Machine Unlearning Evaluation in Diffusion Modelscited by 47
  25. 2025SoFar: Language-Grounded Orientation Bridges Spatial Reasoning and Object Manipulationcited by 46
  26. 2024A Survey on Transformer Compressioncited by 44
  27. 2026Image Generators are Generalist Vision Learnerscited by 40
  28. 2024Autoregressive Models in Vision: A Surveycited by 39
  29. 2024Image Segmentation in Foundation Model Era: A Surveycited by 35
  30. 2024VLMPC: Vision-Language Model Predictive Control for Robotic Manipulationcited by 32
  31. 2024How Far Are We From AGI: Are LLMs All We Need?cited by 31
  32. 2024PixWizard: Versatile Image-to-Image Visual Assistant with Open-Language Instructionscited by 31
  33. 2024PDE Generalization of In-Context Operator Networks: A Study on 1D Scalar Nonlinear Conservation Lawscited by 30
  34. 2024Scaling 4D Representationscited by 30
  35. 2024Vision Superalignment: Weak-to-Strong Generalization for Vision Foundation Modelscited by 28
  36. 2024Segment Anything without Supervisioncited by 27
  37. 2024MarS: a Financial Market Simulation Engine Powered by Generative Foundation Modelcited by 27
  38. 2024CrossEarth: Geospatial Vision Foundation Model for Domain Generalizable Remote Sensing Semantic Segmentationcited by 27

50 most cited of 212 All 212 (JSON)