mlsanityData to 14 Oct 2026

NExT-GPT: Any-to-Any Multimodal LLM

arXiv 2309.05519 ↗First posted 11 Sep 2023cs.AI cs.CL cs.LGcited by 615

On the map: Multimodal LLMs / vision-language models → MLLM training and visual instruction tuning

Abstract (arXiv)

While recently Multimodal Large Language Models (MM-LLMs) have made exciting strides, they mostly fall prey to the limitation of only input-side multimodal understanding, without the ability to produce content in multiple modalities. As we humans always perceive the world and communicate with people through various modalities, developing any-to-any MM-LLMs capable of accepting and delivering content in any modality becomes essential to human-level AI. To fill the gap, we present an end-to-end general-purpose any-to-any MM-LLM system, NExT-GPT. We connect an LLM with multimodal adaptors and different diffusion decoders, enabling NExT-GPT to perceive inputs and generate outputs in arbitrary combinations of text, images, videos, and audio. By leveraging the existing well-trained highly-performing encoders and decoders, NExT-GPT is tuned with only a small amount of parameter (1%) of certain projection layers, which not only benefits low-cost training and also facilitates convenient expansion to more potential modalities. Moreover, we introduce a modality-switching instruction tuning (MosIT) and manually curate a high-quality dataset for MosIT, based on which NExT-GPT is empowered with complex cross-modal semantic understanding and content generation. Overall, our research showcases the promising possibility of building an AI agent capable of modeling universal modalities, paving the way for more human-like AI research in the community. Project page: https://next-gpt.github.io/

Key facts. Cited by 615 papers in the map (data through 14 Oct 2026). Placed in MLLM training and visual instruction tuning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 112 of its 136 references that are in the map. First posted on arXiv 11 Sep 2023.

Builds on

112 of 136 references.

Closest ancestors · 9

Same direction, newest first

  1. 2023StableLLaVA: Enhanced Visual Instruction Tuning with Synthesized Image-Dialogue Data20 Aug
  2. 2023SEED-Bench: Benchmarking Multimodal LLMs with Generative Comprehension30 Jul
  3. 2023ChatSpot: Bootstrapping Multimodal LLMs via Precise Referring Instruction Tuning18 Jul
  4. 2023BuboGPT: Enabling Visual Grounding in Multi-Modal LLMs17 Jul
  5. 2023MMBench: Is Your Multi-modal Model an All-around Player?12 Jul
  6. 2023Generating Images with Multimodal Language Models26 May
  7. 2023SpeechGPT: Empowering Large Language Models with Intrinsic Cross-Modal Conversational Abilities18 May
  8. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuning11 May
  9. 2023VPGTrans: Transfer Visual Prompt Generator across LLMs2 May

Other references · 103

  1. 2017Attention Is All You Needcited by 60,361
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  3. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  4. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,863
  5. 2023GPT-4 Technical Reportcited by 18,396
  6. 2014Microsoft COCO: Common Objects in Contextcited by 17,608
  7. 2022Training language models to follow instructions with human feedbackcited by 15,971
  8. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
  9. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  10. 2023Visual Instruction Tuningcited by 7,869
  11. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,137
  12. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,386
38 more
  1. 2023SDXL: Improving Latent Diffusion Models for High-Resolution Image Synthesiscited by 4,356
  2. 2021Zero-Shot Text-to-Image Generationcited by 4,288
  3. 2022LAION-5B: An open large-scale dataset for training next generation image-text modelscited by 4,152
  4. 2021GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Modelscited by 3,614
  5. 2022DreamBooth: Fine Tuning Text-to-Image Diffusion Models for Subject-Driven Generationcited by 3,300
  6. 2022Scaling Instruction-Finetuned Language Modelscited by 3,024
  7. 2023MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Modelscited by 2,711
  8. 2016Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotationscited by 2,656
  9. 2018Large Scale GAN Training for High Fidelity Natural Image Synthesiscited by 2,626
  10. 2020Learning to summarize from human feedbackcited by 2,588
  11. 2016Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answeringcited by 2,516
  12. 2022Prompt-to-Prompt Image Editing with Cross Attention Controlcited by 2,431
  13. 2021HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Unitscited by 2,342
  14. 2022An Image is Worth One Word: Personalizing Text-to-Image Generation using Textual Inversioncited by 2,245
  15. 2021SDEdit: Guided Image Synthesis and Editing with Stochastic Differential Equationscited by 1,937
  16. 2022Make-A-Video: Text-to-Video Generation without Text-Video Datacited by 1,742
  17. 2023Video-LLaMA: An Instruction-tuned Audio-Visual Language Model for Video Understandingcited by 1,471
  18. 2019Generating Diverse High-Fidelity Images with VQ-VAE-2cited by 1,387
  19. 2023T2I-Adapter: Learning Adapters to Dig out More Controllable Ability for Text-to-Image Diffusion Modelscited by 1,351
  20. 2017Bottom-Up and Top-Down Attention for Image Captioning and Visual Question Answeringcited by 1,312
  21. 2022Scaling Autoregressive Models for Content-Rich Text-to-Image Generationcited by 1,295
  22. 2021Conceptual 12M: Pushing Web-Scale Image-Text Pre-Training To Recognize Long-Tail Visual Conceptscited by 1,216
  23. 2020Oscar: Object-Semantics Aligned Pre-training for Vision-Language Taskscited by 1,214
  24. 2023HuggingGPT: Solving AI Tasks with ChatGPT and its Friends in Hugging Facecited by 1,190
  25. 2021Frozen in Time: A Joint Video and Image Encoder for End-to-End Retrievalcited by 1,177
  26. 2023ImageBind: One Embedding Space To Bind Them Allcited by 1,175
  27. 2019OK-VQA: A Visual Question Answering Benchmark Requiring External Knowledgecited by 1,121
  28. 2023Video-ChatGPT: Towards Detailed Video Understanding via Large Vision and Language Modelscited by 1,062
  29. 2023mPLUG-Owl: Modularization Empowers Large Language Models with Multimodalitycited by 1,040
  30. 2022CogVideo: Large-scale Pretraining for Text-to-Video Generation via Transformerscited by 985
  31. 2014Deep Visual-Semantic Alignments for Generating Image Descriptionscited by 968
  32. 2022Tune-A-Video: One-Shot Tuning of Image Diffusion Models for Text-to-Video Generationcited by 961
  33. 2018VizWiz Grand Challenge: Answering Visual Questions from Blind Peoplecited by 858
  34. 2023VideoChat: Chat-Centric Video Understandingcited by 857
  35. 2022OFA: Unifying Architectures, Tasks, and Modalities Through a Simple Sequence-to-Sequence Learning Frameworkcited by 745
  36. 2021NExT-QA:Next Phase of Question-Answering to Explaining Temporal Actionscited by 744
  37. 2021CogView: Mastering Text-to-Image Generation via Transformerscited by 694
  38. 2023Visual ChatGPT: Talking, Drawing and Editing with Visual Foundation Modelscited by 668

50 most cited of 103 All 103 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 54; 2024: 254; 2025: 209; 2026: 98.

615 papers.

Sort:

By area:

  1. 2023InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Taskscited by 2,565
  2. 2024How Far Are We to GPT-4V? Closing the Gap to Commercial Multimodal Models with Open-Source Suitescited by 1,050
  3. 2024Show-o: One Single Transformer to Unify Multimodal Understanding and Generationcited by 752
  4. 2023Qwen-Audio: Advancing Universal Audio Understanding via Unified Large-Scale Audio-Language Modelscited by 608
  5. 2024Qwen2-Audio Technical Reportcited by 560
  6. 2024Large Language Models: A Surveycited by 525
  7. 2024Janus: Decoupling Visual Encoding for Unified Multimodal Understanding and Generationcited by 457
  8. 2024OmniGen: Unified Image Generationcited by 394
  9. 2024MLLM-as-a-Judge: Assessing Multimodal LLM-as-a-Judge with Vision-Language Benchmarkcited by 372
  10. 2023Chat-UniVi: Unified Visual Representation Empowers Large Language Models with Image and Video Understandingcited by 371
  11. 2024SEED-X: Multimodal Models with Unified Multi-granularity Comprehension and Generationcited by 331
  12. 20243D-VLA: A 3D Vision-Language-Action Generative World Modelcited by 328
38 more
  1. 2025R1-VL: Learning to Reason with Multimodal Large Language Models via Step-wise Group Relative Policy Optimizationcited by 301
  2. 2024Hunyuan-DiT: A Powerful Multi-Resolution Diffusion Transformer with Fine-Grained Chinese Understandingcited by 298
  3. 2024A Survey on Knowledge Distillation of Large Language Modelscited by 284
  4. 2025MMaDA: Multimodal Large Diffusion Language Modelscited by 250
  5. 2025Show-o2: Improved Native Unified Multimodal Modelscited by 250
  6. 2023Multimodal Foundation Models: From Specialists to General-Purpose Assistantscited by 247
  7. 2025Transfer between Modalities with MetaQueriescited by 236
  8. 2023DriveMLM: Aligning Multi-Modal Large Language Models with Behavioral Planning States for Autonomous Drivingcited by 233
  9. 2024VideoLLM-online: Online Video Large Language Model for Streaming Videocited by 225
  10. 2024G-Retriever: Retrieval-Augmented Generation for Textual Graph Understanding and Question Answeringcited by 223
  11. 2024AnyGPT: Unified Multimodal LLM with Discrete Sequence Modelingcited by 217
  12. 2024MANTIS: Interleaved Multi-Image Instruction Tuningcited by 216
  13. 2023A Survey on Video Diffusion Modelscited by 206
  14. 2024TokenFlow: Unified Image Tokenizer for Multimodal Understanding and Generationcited by 202
  15. 2024HQ-Edit: A High-Quality Dataset for Instruction-based Image Editingcited by 193
  16. 2024Large Language Models for Education: A Survey and Outlookcited by 188
  17. 2025Multimodal Chain-of-Thought Reasoning: A Comprehensive Surveycited by 182
  18. 2024Ovis: Structural Embedding Alignment for Multimodal Large Language Modelcited by 169
  19. 2026Cosmos 3: Omnimodal World Models for Physical AIcited by 165
  20. 2024JanusFlow: Harmonizing Autoregression and Rectified Flow for Unified Multimodal Understanding and Generationcited by 163
  21. 2024Video-of-Thought: Step-by-Step Video Reasoning from Perception to Cognitioncited by 153
  22. 2025UniTok: A Unified Tokenizer for Visual Generation and Understandingcited by 153
  23. 2024Lumina-mGPT: Illuminate Flexible Photorealistic Text-to-Image Generation with Multimodal Generative Pretrainingcited by 150
  24. 2024SEED-Bench-2-Plus: Benchmarking Multimodal Large Language Models with Text-Rich Visual Comprehensioncited by 146
  25. 2024The Revolution of Multimodal Large Language Models: A Surveycited by 137
  26. 2024ShapeLLM: Universal 3D Object Understanding for Embodied Interactioncited by 137
  27. 2024ChartX & ChartVLM: A Versatile Benchmark and Foundation Model for Complicated Chart Reasoningcited by 128
  28. 2024OMG-LLaVA: Bridging Image-level, Object-level, Pixel-level Reasoning and Understandingcited by 127
  29. 2025Advances and Challenges in Foundation Agents: From Brain-Inspired Intelligence to Evolutionary, Collaborative, and Safe Systemscited by 126
  30. 2024LITA: Language Instructed Temporal-Localization Assistantcited by 125
  31. 2024Empowering Biomedical Discovery with AI Agentscited by 118
  32. 2025HealthGPT: A Medical Large Vision-Language Model for Unifying Comprehension and Generation via Heterogeneous Knowledge Adaptationcited by 114
  33. 2025X-Omni: Reinforcement Learning Makes Discrete Autoregressive Image Generative Models Great Againcited by 112
  34. 2023Large Models for Time Series and Spatio-Temporal Data: A Survey and Outlookcited by 110
  35. 2024Uni-MoE: Scaling Unified Multimodal LLMs with Mixture of Expertscited by 108
  36. 2024On-Device Language Models: A Comprehensive Reviewcited by 104
  37. 2024LongLLaVA: Scaling Multi-modal LLMs to 1000 Images Efficiently via a Hybrid Architecturecited by 102
  38. 2024GroundingGPT:Language Enhanced Multi-modal Grounding Modelcited by 101

50 most cited of 615 All 615 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025Empowering Multimodal LLMs with External Tools: A Comprehensive Survey
  2. 2023X-LLM: Bootstrapping Advanced Large Language Models by Treating Multi-Modalities as Foreign Languages
  3. 2023The Dawn of LMMs: Preliminary Explorations with GPT-4V(ision)
  4. 2025Towards LLM-Centric Multimodal Fusion: A Survey on Integration Strategies and Techniques
  5. 2025On Path to Multimodal Generalist: General-Level and General-Bench
  6. 2023On the Performance of Multimodal Language Models
  7. 2024Multi-modal Instruction Tuned LLMs with Fine-grained Visual Perception
  8. 2026Cost-Efficient Multimodal LLM Inference via Cross-Tier GPU Heterogeneity