mlsanityData to 7 Oct 2026

Multi-modal Auto-regressive Modeling via Visual Words

arXiv 2403.07720 ↗First posted 12 Mar 2024cs.CV cs.AIcited by 4

On the map: LLM reasoning → Multimodal reasoning

Abstract (arXiv)

Large Language Models (LLMs), benefiting from the auto-regressive modelling approach performed on massive unannotated texts corpora, demonstrates powerful perceptual and reasoning capabilities. However, as for extending auto-regressive modelling to multi-modal scenarios to build Large Multi-modal Models (LMMs), there lies a great difficulty that the image information is processed in the LMM as continuous visual embeddings, which cannot obtain discrete supervised labels for classification.In this paper, we successfully perform multi-modal auto-regressive modeling with a unified objective for the first time.Specifically, we propose the concept of visual tokens, which maps the visual features to probability distributions over LLM's vocabulary, providing supervision information for visual modelling.We further explore the distribution of visual features in the semantic space within LMM and the possibility of using text embeddings to represent visual information.Experimental results and ablation studies on 5 VQA tasks and 4 benchmark toolkits validate the powerful performance of our proposed approach.

Key facts. Cited by 4 papers in the map (data through 7 Oct 2026). Placed in Multimodal reasoning, one of 17 directions in LLM reasoning. Builds on 20 of its 30 references that are in the map. First posted on arXiv 12 Mar 2024.

Builds on

20 of 30 references.

Closest ancestors · 3

Same direction, newest first

  1. 2023Unified Language-Vision Pretraining in LLM with Dynamic Discrete Visual Tokenization9 Sep
  2. 2023MMBench: Is Your Multi-modal Model an All-around Player?12 Jul
  3. 2022Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question Answering20 Sep

Other references · 17

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generationcited by 4,310
  3. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuningcited by 2,855
  4. 2016Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answeringcited by 2,498
  5. 2019Towards VQA Models That Can Readcited by 1,696
  6. 2023Evaluating Object Hallucination in Large Vision-Language Modelscited by 1,696
  7. 2021Align before Fuse: Vision and Language Representation Learning with Momentum Distillationcited by 1,612
  8. 2023MM-Vet: Evaluating Large Multimodal Models for Integrated Capabilitiescited by 1,221
  9. 2023mPLUG-Owl: Modularization Empowers Large Language Models with Multimodalitycited by 1,038
  10. 2023Unified-IO 2: Scaling Autoregressive Multimodal Models with Vision, Language, Audio, and Actioncited by 301
  11. 2024MoE-LLaVA: Mixture of Experts for Large Vision-Language Modelscited by 298
  12. 2023DreamLLM: Synergistic Multimodal Comprehension and Creationcited by 296
5 more
  1. 2023MultiModal-GPT: A Vision and Language Model for Dialogue with Humanscited by 255
  2. 2023Sequential Modeling Enables Scalable Learning for Large Vision Modelscited by 212
  3. 2023Emu: Generative Pretraining in Multimodalitycited by 156
  4. 2024MM-Interleaved: Interleaved Image-Text Generative Modeling via Multi-modal Feature Synchronizercited by 80
  5. 2023VL-GPT: A Generative Pre-trained Transformer for Vision and Language Understanding and Generationcited by 41

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 2; 2025: 0; 2026: 2.

4 papers.

  1. 2024Towards Unifying Understanding and Generation in the Era of Vision Foundation Models: A Survey from the Autoregression Perspectivecited by 10
  2. 2026EvoComp: Learning Visual Token Compression for Multimodal Large Language Models via Semantic-Guided Evolutionary Labelingcited by 4
  3. 2024VHASR: A Multimodal Speech Recognition System With Vision Hotwordscited by 2
  4. 2026DenseMLLM: Standard Multimodal LLMs for Dense Predictioncited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Growing Visual Generative Capacity for Pre-Trained MLLMs
  2. 2024Efficient Large Multi-modal Models via Visual Context Compression
  3. 2026UVU: Improving Multimodal Understanding via Vision-Language Unified Autoregressive Paradigm
  4. 2023MLLMs-Augmented Visual-Language Representation Learning
  5. 2025Unleashing the Intrinsic Visual Representation Capability of Multimodal Large Language Models
  6. 2026What Do Visual Tokens Really Encode? Uncovering Sparsity and Redundancy in Multimodal Large Language Models
  7. 2026Analyzing Diffusion and Autoregressive Vision Language Models in Multimodal Embedding Space
  8. 2024Finer: Investigating and Enhancing Fine-Grained Visual Concept Recognition in Large Vision Language Models