mlsanityData to 14 Oct 2026

AnyGPT: Unified Multimodal LLM with Discrete Sequence Modeling

arXiv 2402.12226 ↗First posted 19 Feb 2024cs.CL cs.AI cs.CVcited by 217

On the map: Multimodal LLMs / vision-language models → MLLM training and visual instruction tuning

Abstract (arXiv)

We introduce AnyGPT, an any-to-any multimodal language model that utilizes discrete representations for the unified processing of various modalities, including speech, text, images, and music. AnyGPT can be trained stably without any alterations to the current large language model (LLM) architecture or training paradigms. Instead, it relies exclusively on data-level preprocessing, facilitating the seamless integration of new modalities into LLMs, akin to the incorporation of new languages. We build a multimodal text-centric dataset for multimodal alignment pre-training. Utilizing generative models, we synthesize the first large-scale any-to-any multimodal instruction dataset. It consists of 108k samples of multi-turn conversations that intricately interweave various modalities, thus equipping the model to handle arbitrary combinations of multimodal inputs and outputs. Experimental results demonstrate that AnyGPT is capable of facilitating any-to-any multimodal conversation while achieving performance comparable to specialized models across all modalities, proving that discrete representations can effectively and conveniently unify multiple modalities within a language model. Demos are shown in https://junzhan2000.github.io/AnyGPT.github.io/

Key facts. Cited by 217 papers in the map (data through 14 Oct 2026). Placed in MLLM training and visual instruction tuning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 46 of its 57 references that are in the map. First posted on arXiv 19 Feb 2024.

Builds on

46 of 57 references.

Closest ancestors · 8

Same direction, newest first

  1. 2023Generative Multimodal Models are In-Context Learners20 Dec
  2. 2023CoDi-2: In-Context, Interleaved, and Interactive Any-to-Any Generation30 Nov
  3. 2023DreamLLM: Synergistic Multimodal Comprehension and Creation20 Sep
  4. 2023NExT-GPT: Any-to-Any Multimodal LLM11 Sep
  5. 2023JourneyDB: A Benchmark for Generative Image Understanding3 Jul
  6. 2023Generating Images with Multimodal Language Models26 May
  7. 2023SpeechGPT: Empowering Large Language Models with Intrinsic Cross-Modal Conversational Abilities18 May
  8. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuning11 May

Other references · 38

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  2. 2020Language Models are Few-Shot Learnerscited by 32,419
  3. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  4. 2015U-Net: Convolutional Networks for Biomedical Image Segmentationcited by 19,713
  5. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,863
  6. 2023GPT-4 Technical Reportcited by 18,396
  7. 2014Microsoft COCO: Common Objects in Contextcited by 17,608
  8. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  9. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  10. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,137
  11. 2017Neural Discrete Representation Learningcited by 4,712
  12. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,386
26 more
  1. 2022LAION-5B: An open large-scale dataset for training next generation image-text modelscited by 4,152
  2. 2022Robust Speech Recognition via Large-Scale Weak Supervisioncited by 4,057
  3. 2020wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representationscited by 3,905
  4. 2022Self-Instruct: Aligning Language Models with Self-Generated Instructionscited by 2,724
  5. 2019Common Voice: A Massively-Multilingual Speech Corpuscited by 1,384
  6. 2021SoundStream: An End-to-End Neural Audio Codeccited by 771
  7. 2023Neural Codec Language Models are Zero-Shot Text to Speech Synthesizerscited by 752
  8. 2022Large-scale Contrastive Language-Audio Pretraining with Feature Fusion and Keyword-to-Caption Augmentationcited by 719
  9. 2022High Fidelity Neural Audio Compressioncited by 687
  10. 2022AudioLM: a Language Modeling Approach to Audio Generationcited by 616
  11. 2023VideoPoet: A Large Language Model for Zero-Shot Video Generationcited by 492
  12. 2023Simple and Controllable Music Generationcited by 488
  13. 2020MLS: A Large-Scale Multilingual Dataset for Speech Researchcited by 476
  14. 2015Deep Speech 2: End-to-End Speech Recognition in English and Mandarincited by 451
  15. 2023MusicLM: Generating Music From Textcited by 448
  16. 2021GigaSpeech: An Evolving, Multi-domain ASR Corpus with 10,000 Hours of Transcribed Audiocited by 399
  17. 2023Make-An-Audio: Text-To-Audio Generation with Prompt-Enhanced Diffusion Modelscited by 324
  18. 2023Unified-IO 2: Scaling Autoregressive Multimodal Models with Vision, Language, Audio, and Actioncited by 302
  19. 2023Multimodal C4: An Open, Billion-scale Corpus of Images Interleaved with Textcited by 208
  20. 2023Any-to-Any Generation via Composable Diffusioncited by 204
  21. 2023Making LLaMA SEE and Draw with SEED Tokenizercited by 185
  22. 2023Emu: Generative Pretraining in Multimodalitycited by 157
  23. 2023Planting a SEED of Vision in Large Language Modelcited by 140
  24. 2023SoundStorm: Efficient Parallel Audio Generationcited by 114
  25. 2023SpeechTokenizer: Unified Speech Tokenizer for Speech Large Language Modelscited by 87
  26. 2023LLark: A Multimodal Instruction-Following Language Model for Musiccited by 29

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 75; 2025: 94; 2026: 48.

217 papers.

Sort:

By area:

  1. 2025Qwen2.5-Omni Technical Reportcited by 736
  2. 2024Mini-Gemini: Mining the Potential of Multi-modality Vision Language Modelscited by 365
  3. 2024VILA-U: a Unified Foundation Model Integrating Visual Understanding and Generationcited by 295
  4. 2024MANTIS: Interleaved Multi-Image Instruction Tuningcited by 216
  5. 2025VITA-1.5: Towards GPT-4o Level Real-Time Vision and Speech Interactioncited by 197
  6. 2025Multimodal Chain-of-Thought Reasoning: A Comprehensive Surveycited by 182
  7. 2024VITA: Towards Open-Source Interactive Omni Multimodal LLMcited by 173
  8. 2024LLaMA-Omni: Seamless Speech Interaction with Large Language Modelscited by 173
  9. 2024VisionLLM v2: An End-to-End Generalist Multimodal Large Language Model for Hundreds of Vision-Language Taskscited by 137
  10. 2024WavTokenizer: an Efficient Acoustic Discrete Codec Tokenizer for Audio Language Modelingcited by 129
  11. 2025Advances and Challenges in Foundation Agents: From Brain-Inspired Intelligence to Evolutionary, Collaborative, and Safe Systemscited by 126
  12. 2025On The Landscape of Spoken Language Models: A Comprehensive Surveycited by 118
38 more
  1. 2025MMAR: A Challenging Benchmark for Deep Reasoning in Speech, Audio, Music, and Their Mixcited by 117
  2. 2025A Comprehensive Survey in LLM(-Agent) Full Stack Safety: Data, Training and Deploymentcited by 116
  3. 2024ANOLE: An Open, Autoregressive, Native Large Multimodal Models for Interleaved Image-Text Generationcited by 114
  4. 2025Baichuan-Omni-1.5 Technical Reportcited by 111
  5. 2024Uni-MoE: Scaling Unified Multimodal LLMs with Mixture of Expertscited by 108
  6. 2024OmniBench: Towards The Future of Universal Omni-Language Modelscited by 102
  7. 2025ShareGPT-4o-Image: Aligning Multimodal Models with GPT-4o-Level Image Generationcited by 97
  8. 2024Deep Multimodal Learning with Missing Modality: A Surveycited by 92
  9. 2025Perception, Reason, Think, and Plan: A Survey on Large Multimodal Reasoning Modelscited by 90
  10. 2024Recent Advances in Speech Language Models: A Surveycited by 87
  11. 2025Unified Multimodal Understanding and Generation Models: Advances, Challenges, and Opportunitiescited by 85
  12. 2025Vision-Language-Action Models for Robotics: A Review Towards Real-World Applicationscited by 84
  13. 2024ILLUME: Illuminating Your LLMs to See, Draw, and Self-Enhancecited by 64
  14. 2024A Comprehensive Review of Multimodal Large Language Models: Performance and Challenges Across Different Taskscited by 57
  15. 2024The Curse of Multi-Modalities: Evaluating Hallucinations of Large Multimodal Models across Language, Visual, and Audiocited by 53
  16. 2025FUDOKI: Discrete Flow-based Unified Understanding and Generation via Kinetic-Optimal Velocitiescited by 46
  17. 2024EMOVA: Empowering Language Models to See, Hear and Speak with Vivid Emotionscited by 45
  18. 2024MUSE-VL: Modeling Unified VLM through Semantic Discrete Encodingcited by 44
  19. 2025Latent Sketchpad: Sketching Visual Thoughts to Elicit Multimodal Reasoning in MLLMscited by 43
  20. 2024Baichuan-Omni Technical Reportcited by 42
  21. 2025EVEv2: Improved Baselines for Encoder-Free Vision-Language Modelscited by 41
  22. 2024Graphic Design with Large Multimodal Modelcited by 40
  23. 2024M3GPT: An Advanced Multimodal, Multitask Framework for Motion Comprehension and Generationcited by 38
  24. 2024Foundation Models for Music: A Surveycited by 37
  25. 2024LLMs Meet Multimodal Generation and Editing: A Surveycited by 35
  26. 2025MeViS: A Multi-Modal Dataset for Referring Motion Expression Video Segmentationcited by 35
  27. 2024DASB - Discrete Audio and Speech Benchmarkcited by 32
  28. 2024Understanding Sounds, Missing the Questions: The Challenge of Object Hallucination in Large Audio-Language Modelscited by 31
  29. 2024Causal Diffusion Transformers for Generative Modelingcited by 31
  30. 2024Auto-Encoding Morph-Tokens for Multimodal LLMcited by 30
  31. 2024CoMT: A Novel Benchmark for Chain of Multi-modal Thought on Large Vision-Language Modelscited by 30
  32. 2025AVCD: Mitigating Hallucinations in Audio-Visual Large Language Models through Contrastive Decodingcited by 30
  33. 2024AutoGeo: Automating Geometric Image Dataset Creation for Enhanced Geometry Understandingcited by 29
  34. 2025InteractiveOmni: A Unified Omni-modal Model for Audio-Visual Multi-turn Dialoguecited by 29
  35. 2024Reason3D: Searching and Reasoning 3D Segmentation via Large Language Modelcited by 28
  36. 2024Holistic Evaluation for Interleaved Text-and-Image Generationcited by 28
  37. 2024EventGPT: Event Stream Understanding with Multimodal Large Language Modelscited by 28
  38. 2024Next Token Prediction Towards Multimodal Intelligence: A Comprehensive Surveycited by 27

50 most cited of 217 All 217 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2023AnyMAL: An Efficient and Scalable Any-Modality Augmented Language Model
  2. 2023X-LLM: Bootstrapping Advanced Large Language Models by Treating Multi-Modalities as Foreign Languages
  3. 2023On the Performance of Multimodal Language Models
  4. 2026Alignment Is All You Need: Instruction-Free Training for General Audio-Language Models
  5. 2023Macaw-LLM: Multi-Modal Language Modeling with Image, Audio, Video, and Text Integration
  6. 2023OneLLM: One Framework to Align All Modalities with Language
  7. 2023ChatBridge: Bridging Modalities with Large Language Model as a Language Catalyst
  8. 2024The Revolution of Multimodal Large Language Models: A Survey