mlsanityData to 7 Oct 2026

Next Token Prediction Towards Multimodal Intelligence: A Comprehensive Survey

arXiv 2412.18619 ↗First posted 16 Dec 2024cs.CL cs.AI cs.CVcited by 27

Not on the map. Why?

Abstract (arXiv)

Building on the foundations of language modeling in natural language processing, Next Token Prediction (NTP) has evolved into a versatile training objective for machine learning tasks across various modalities, achieving considerable success. As Large Language Models (LLMs) have advanced to unify understanding and generation tasks within the textual modality, recent research has shown that tasks from different modalities can also be effectively encapsulated within the NTP framework, transforming the multimodal information into tokens and predict the next one given the context. This survey introduces a comprehensive taxonomy that unifies both understanding and generation within multimodal learning through the lens of NTP. The proposed taxonomy covers five key aspects: Multimodal tokenization, MMNTP model architectures, unified task representation, datasets & evaluation, and open challenges. This new taxonomy aims to aid researchers in their exploration of multimodal intelligence. An associated GitHub repository collecting the latest papers and repos is available at https://github.com/LMM101/Awesome-Multimodal-Next-Token-Prediction

Key facts. Cited by 27 papers in the map (data through 7 Oct 2026). Builds on 334 of its 518 references that are in the map. First posted on arXiv 16 Dec 2024.

Builds on

334 of 518 references.

References · 334

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  3. 2020Denoising Diffusion Probabilistic Modelscited by 20,851
  4. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  5. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  6. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  7. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,082
  8. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  9. 2021Diffusion Models Beat GANs on Image Synthesiscited by 8,185
  10. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,916
  11. 2023Visual Instruction Tuningcited by 7,792
  12. 2021Masked Autoencoders Are Scalable Vision Learnerscited by 7,339
38 more
  1. 2021Emerging Properties in Self-Supervised Vision Transformerscited by 6,893
  2. 2020Scaling Laws for Neural Language Modelscited by 6,411
  3. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,069
  4. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,767
  5. 2017Neural Discrete Representation Learningcited by 4,650
  6. 2023SDXL: Improving Latent Diffusion Models for High-Resolution Image Synthesiscited by 4,290
  7. 2021Zero-Shot Text-to-Image Generationcited by 4,276
  8. 2024Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolutioncited by 4,151
  9. 2013Auto-Encoding Variational Bayescited by 3,980
  10. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,611
  11. 2022A ConvNet for the 2020scited by 3,545
  12. 2024Gemini 1.5: Unlocking multimodal understanding across millions of tokens of contextcited by 3,175
  13. 2023Sigmoid Loss for Language Image Pre-Trainingcited by 3,048
  14. 2023RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Controlcited by 2,993
  15. 2022Training Compute-Optimal Large Language Modelscited by 2,927
  16. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuningcited by 2,855
  17. 2023Diffusion Policy: Visuomotor Policy Learning via Action Diffusioncited by 2,730
  18. 2023MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Modelscited by 2,689
  19. 2016Perceptual Losses for Real-Time Style Transfer and Super-Resolutioncited by 2,678
  20. 2023A Survey of Large Language Modelscited by 2,537
  21. 2023InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Taskscited by 2,534
  22. 2022Prompt-to-Prompt Image Editing with Cross Attention Controlcited by 2,409
  23. 2021BEiT: BERT Pre-Training of Image Transformerscited by 2,360
  24. 2020The Pile: An 800GB Dataset of Diverse Text for Language Modelingcited by 2,346
  25. 2023Qwen-VL: A Versatile Vision-Language Model for Understanding, Localization, Text Reading, and Beyondcited by 2,179
  26. 2023MMBench: Is Your Multi-modal Model an All-around Player?cited by 2,105
  27. 2023MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGIcited by 2,044
  28. 2023PaLM-E: An Embodied Multimodal Language Modelcited by 2,037
  29. 2013Estimating or Propagating Gradients Through Stochastic Neurons for Conditional Computationcited by 2,031
  30. 2024Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phonecited by 1,883
  31. 2023ToolLLM: Facilitating Large Language Models to Master 16000+ Real-world APIscited by 1,880
  32. 2020Conformer: Convolution-augmented Transformer for Speech Recognitioncited by 1,838
  33. 2023VBench: Comprehensive Benchmark Suite for Video Generative Modelscited by 1,755
  34. 2016WaveNet: A Generative Model for Raw Audiocited by 1,710
  35. 2023Evaluating Object Hallucination in Large Vision-Language Modelscited by 1,696
  36. 2023MME: A Comprehensive Evaluation Benchmark for Multimodal Large Language Modelscited by 1,624
  37. 2021ViViT: A Video Vision Transformercited by 1,610
  38. 2023MathVista: Evaluating Mathematical Reasoning of Foundation Models in Visual Contextscited by 1,582

50 most cited of 334 All 334 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 1; 2025: 19; 2026: 7.

27 papers.

Sort:

By area:

  1. 2025WISE: A World Knowledge-Informed Semantic Evaluation for Text-to-Image Generationcited by 196
  2. 2025Thinking with Images for Multimodal Reasoning: Foundations, Methods, and Future Frontierscited by 193
  3. 2025On The Landscape of Spoken Language Models: A Comprehensive Surveycited by 116
  4. 2024Parallelized Autoregressive Visual Generationcited by 58
  5. 2025Selftok: Discrete Visual Tokens of Autoregression, by Diffusion, and for Reasoningcited by 31
  6. 2025Bridging Continuous and Discrete Tokens for Autoregressive Visual Generationcited by 29
  7. 2025Next Block Prediction: Video Generation via Semi-Autoregressive Modelingcited by 26
  8. 2025L-MTP: Leap Multi-Token Prediction Beyond Adjacent Context for Large Language Modelscited by 14
  9. 2025Token Communication in the Era of Large Models: An Information Bottleneck-Based Approachcited by 10
  10. 2025Discrete Tokenization for Multimodal LLMs: A Comprehensive Surveycited by 7
  11. 2025ViLaD: A Large Vision Language Diffusion Framework for End-to-End Autonomous Drivingcited by 7
  12. 2025Region-Aware Multimodal Large Language Model via SlowFast Tokenization and Pseudo-Mask Guidance for 3D CT Report Generationcited by 6
15 more
  1. 2026Toward Native Multimodal Modeling: A Roadmapcited by 6
  2. 2025Toward Clinically Explainable AI for Medical Diagnosis: A Foundation Model with Human-Compatible Reasoning via Reinforcement Learningcited by 5
  3. 2025Have we unified image generation and understanding yet? An empirical study of GPT-4o's image generation abilitycited by 4
  4. 2025Image Tokens Matter: Mitigating Hallucination in Discrete Tokenizer-based Large Vision-Language Models via Latent Editingcited by 4
  5. 2026NITP: Next Implicit Token Prediction for LLM Pre-trainingcited by 3
  6. 2025Post-Training and Test-Time Scaling of Generative Agent Behavior Models for Interactive Autonomous Drivingcited by 2
  7. 2025Adapting Pretrained Language Models for Citation Classification via Self-Supervised Contrastive Learningcited by 1
  8. 2025MedErr-CT: A Visual Question Answering Benchmark for Identifying and Correcting Errors in CT Reportscited by 1
  9. 2025A Unified Low-level Foundation Model for Enhancing Pathology Image Qualitycited by 1
  10. 2025Alternatives To Next Token Prediction In Text Generation -- A Surveycited by 1
  11. 2026SimReg: Achieving Higher Performance in the Pretraining via Embedding Similarity Regularizationcited by 0
  12. 2026PathAR: Structure-First Autoregressive Synthesis of Multimodal Pathology Imagescited by 0
  13. 2026MotionMAR: Multi-scale Auto-Regressive Human Motion Reconstruction from Sparse Observationscited by 0
  14. 2026SAGG: Sample-Adaptive Gradient Gating for Robust Multimodal Learning under Heterogeneous Corruptioncited by 0
  15. 2026Mutually Adversarial Self-Training with Evolving Data for Unified Multimodal Modelscited by 0