mlsanityData to 7 Oct 2026

Multimodal Autoregressive Pre-training of Large Vision Encoders

arXiv 2411.14402 ↗First posted 21 Nov 2024cs.CV cs.LGcited by 108

Not on the map. Why?

Cited by 42 in Multimodal LLMs / vision-language models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

We introduce a novel method for pre-training of large-scale vision encoders. Building on recent advancements in autoregressive pre-training of vision models, we extend this framework to a multimodal setting, i.e., images and text. In this paper, we present AIMV2, a family of generalist vision encoders characterized by a straightforward pre-training process, scalability, and remarkable performance across a range of downstream tasks. This is achieved by pairing the vision encoder with a multimodal decoder that autoregressively generates raw image patches and text tokens. Our encoders excel not only in multimodal evaluations but also in vision benchmarks such as localization, grounding, and classification. Notably, our AIMV2-3B encoder achieves 89.5% accuracy on ImageNet-1k with a frozen trunk. Furthermore, AIMV2 consistently outperforms state-of-the-art contrastive models (e.g., CLIP, SigLIP) in multimodal image understanding across diverse settings.

Key facts. Cited by 108 papers in the map (data through 7 Oct 2026). Builds on 107 of its 131 references that are in the map. First posted on arXiv 21 Nov 2024.

Builds on

107 of 131 references.

References · 107

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  3. 2020Language Models are Few-Shot Learnerscited by 32,199
  4. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  5. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  6. 2023GPT-4 Technical Reportcited by 18,132
  7. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  8. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  9. 2024The Llama 3 Herd of Modelscited by 14,192
  10. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  11. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  12. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 11,979
38 more
  1. 2020End-to-End Object Detection with Transformerscited by 7,384
  2. 2021Masked Autoencoders Are Scalable Vision Learnerscited by 7,339
  3. 2017Mask R-CNNcited by 7,259
  4. 2021Emerging Properties in Self-Supervised Vision Transformerscited by 6,893
  5. 2020Scaling Laws for Neural Language Modelscited by 6,411
  6. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,069
  7. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
  8. 2016SGDR: Stochastic Gradient Descent with Warm Restartscited by 4,993
  9. 2020Bootstrap your own latent: A new approach to self-supervised Learningcited by 4,712
  10. 2023Improved Baselines with Visual Instruction Tuningcited by 4,362
  11. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,337
  12. 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generationcited by 4,310
  13. 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,848
  14. 2023Grounding DINO: Marrying DINO with Grounded Pre-Training for Open-Set Object Detectioncited by 3,274
  15. 2023Qwen Technical Reportcited by 3,273
  16. 2020Taming Transformers for High-Resolution Image Synthesiscited by 3,229
  17. 2024Gemini 1.5: Unlocking multimodal understanding across millions of tokens of contextcited by 3,175
  18. 2023Sigmoid Loss for Language Image Pre-Trainingcited by 3,048
  19. 2022Training Compute-Optimal Large Language Modelscited by 2,927
  20. 2020Unsupervised Learning of Visual Features by Contrasting Cluster Assignmentscited by 2,845
  21. 2023MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Modelscited by 2,689
  22. 2016Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answeringcited by 2,498
  23. 2021BEiT: BERT Pre-Training of Image Transformerscited by 2,360
  24. 2013Describing Textures in the Wildcited by 2,110
  25. 2022Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question Answeringcited by 1,881
  26. 2018SlowFast Networks for Video Recognitioncited by 1,862
  27. 2020GLU Variants Improve Transformercited by 1,724
  28. 2019Towards VQA Models That Can Readcited by 1,696
  29. 2023MME: A Comprehensive Evaluation Benchmark for Multimodal Large Language Modelscited by 1,624
  30. 2021Align before Fuse: Vision and Language Representation Learning with Momentum Distillationcited by 1,612
  31. 2018Unsupervised Representation Learning by Predicting Image Rotationscited by 1,586
  32. 2021LAION-400M: Open Dataset of CLIP-Filtered 400 Million Image-Text Pairscited by 1,544
  33. 2017EuroSAT: A Novel Dataset and Deep Learning Benchmark for Land Use and Land Cover Classificationcited by 1,488
  34. 2022ChartQA: A Benchmark for Question Answering about Charts with Visual and Logical Reasoningcited by 1,449
  35. 2015Microsoft COCO Captions: Data Collection and Evaluation Servercited by 1,447
  36. 2019Root Mean Square Layer Normalizationcited by 1,425
  37. 2015Show, Attend and Tell: Neural Image Caption Generation with Visual Attentioncited by 1,421
  38. 2016Unsupervised Learning of Visual Representations by Solving Jigsaw Puzzlescited by 1,379

50 most cited of 107 All 107 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 65; 2026: 43.

108 papers.

Sort:

By area:

  1. 2025V-JEPA 2: Self-Supervised Video Models Enable Understanding, Prediction and Planningcited by 738
  2. 2025GLM-4.5V and GLM-4.1V-Thinking: Towards Versatile Multimodal Reasoning with Scalable Reinforcement Learningcited by 439
  3. 2025Perception Encoder: The best visual embeddings are not at the output of the networkcited by 315
  4. 2025Layer by Layer: Uncovering Hidden Representations in Language Modelscited by 293
  5. 2025LLaVA-OneVision-1.5: Fully Open Framework for Democratized Multimodal Trainingcited by 233
  6. 2025MM-Spatial: Exploring 3D Spatial Understanding in Multimodal LLMscited by 72
  7. 2025Sparse Autoencoders Learn Monosemantic Features in Vision-Language Modelscited by 70
  8. 2025Vision-Language Models Do Not Understand Negationcited by 69
  9. 2025Ovis-U1 Technical Reportcited by 65
  10. 2025Scaling Language-Free Visual Representation Learningcited by 62
  11. 2025VL-JEPA: Joint Embedding Predictive Architecture for Vision-languagecited by 61
  12. 2025Scaling Laws for Optimal Data Mixturescited by 49
38 more
  1. 2025Scaling Laws for Native Multimodal Modelscited by 47
  2. 2025From Flat to Hierarchical: Extracting Sparse Representations with Matching Pursuitcited by 45
  3. 2025Vision as LoRAcited by 37
  4. 2025Interpreting the linear structure of vision-language model embedding spacescited by 32
  5. 2025DINO in the Room: Leveraging 2D Foundation Models for 3D Segmentationcited by 31
  6. 2026LongCat-Next: Lexicalizing Modalities as Discrete Tokenscited by 30
  7. 2025Scaling Vision Pre-Training to 4K Resolutioncited by 26
  8. 2025Franca: Nested Matryoshka Clustering for Scalable Visual Representation Learningcited by 21
  9. 2025Next-Embedding Prediction Makes Strong Vision Learnerscited by 21
  10. 2025DepthMaster: Taming Diffusion Models for Monocular Depth Estimationcited by 18
  11. 2025Analyzing Finetuning Representation Shift for Multimodal LLMs Steeringcited by 15
  12. 2025OpenVision: A Fully-Open, Cost-Effective Family of Advanced Vision Encoders for Multimodal Learningcited by 15
  13. 2025Object Detection with Multimodal Large Vision-Language Models: An In-depth Reviewcited by 14
  14. 2025SAIL-VL2 Technical Reportcited by 14
  15. 2025Region-based Cluster Discrimination for Visual Representation Learningcited by 13
  16. 2025VISTA: Enhancing Vision-Text Alignment in MLLMs via Cross-Modal Mutual Information Maximizationcited by 12
  17. 2026TIPSv2: Advancing Vision-Language Pretraining with Enhanced Patch-Text Alignmentcited by 12
  18. 2025Visual RAG: Expanding MLLM visual knowledge without fine-tuningcited by 11
  19. 2025Adaptive Audio-Visual Speech Recognition via Matryoshka-Based Multimodal LLMscited by 11
  20. 2025Attention, Please! Revisiting Attentive Probing Through the Lens of Efficiencycited by 11
  21. 2025Scaling and Enhancing LLM-based AVSR: A Sparse Mixture of Projectors Approachcited by 10
  22. 2026OneVision-Encoder: Codec-Aligned Sparsity as a Foundational Principle for Multimodal Intelligencecited by 10
  23. 2026Representation Distribution Matching for One-Step Visual Generationcited by 10
  24. 2025MammothModa2: A Unified AR-Diffusion Framework for Multimodal Understanding and Generationcited by 9
  25. 2026Cross-Modal Redundancy and the Geometry of Vision-Language Embeddingscited by 9
  26. 2025CuRe: Cultural Gaps in the Long Tail of Text-to-Image Systemscited by 8
  27. 2026Mage-VL: An Efficient Codec-Native Streaming Multimodal Foundation Modelcited by 8
  28. 2025Seeing is Understanding: Unlocking Causal Attention into Modality-Mutual Attention for Multimodal LLMscited by 7
  29. 2025OpenVision 2: A Family of Generative Pretrained Visual Encoders for Multimodal Learningcited by 7
  30. 2025Visual Instruction Pretraining for Domain-Specific Foundation Modelscited by 7
  31. 2026Efficient Universal Perception Encodercited by 6
  32. 2025Enhancing Vision Foundation Models via Multimodal Continual Pre-Trainingcited by 5
  33. 2025AVA-Bench: Atomic Visual Ability Benchmark for Vision Foundation Modelscited by 5
  34. 2025LaCo: Efficient Layer-wise Compression of Visual Tokens for Multimodal Large Language Modelscited by 5
  35. 2026The Spatial Blindspot of Vision-Language Modelscited by 5
  36. 2025Perceiving Beyond Language Priors: Enhancing Visual Comprehension and Attention in Multimodal Modelscited by 4
  37. 2025Flash-VL 2B: Optimizing Vision-Language Model Performance for Ultra-Low Latency and High Throughputcited by 4
  38. 2025SAILViT: Towards Robust and Generalizable Visual Backbones for MLLMs via Gradual Feature Refinementcited by 4

50 most cited of 108 All 108 (JSON)