mlsanityData to 14 Oct 2026

SpeechGPT: Empowering Large Language Models with Intrinsic Cross-Modal Conversational Abilities

arXiv 2305.11000 ↗First posted 18 May 2023cs.CLcited by 529

On the map: Multimodal LLMs / vision-language models → MLLM training and visual instruction tuning

Abstract (arXiv)

Multi-modal large language models are regarded as a crucial step towards Artificial General Intelligence (AGI) and have garnered significant interest with the emergence of ChatGPT. However, current speech-language models typically adopt the cascade paradigm, preventing inter-modal knowledge transfer. In this paper, we propose SpeechGPT, a large language model with intrinsic cross-modal conversational abilities, capable of perceiving and generating multi-model content. With discrete speech representations, we first construct SpeechInstruct, a large-scale cross-modal speech instruction dataset. Additionally, we employ a three-stage training strategy that includes modality-adaptation pre-training, cross-modal instruction fine-tuning, and chain-of-modality instruction fine-tuning. The experimental results demonstrate that SpeechGPT has an impressive capacity to follow multi-modal human instructions and highlight the potential of handling multiple modalities with one model. Demos are shown in https://0nutation.github.io/SpeechGPT.github.io/.

Key facts. Cited by 529 papers in the map (data through 14 Oct 2026). Placed in MLLM training and visual instruction tuning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 25 of its 35 references that are in the map. First posted on arXiv 18 May 2023.

Builds on

25 of 35 references.

References · 25

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  2. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  3. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
  4. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  5. 2023Visual Instruction Tuningcited by 7,869
  6. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,326
  7. 2020wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representationscited by 3,905
  8. 2022Self-Instruct: Aligning Language Models with Self-Generated Instructionscited by 2,724
  9. 2021HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Unitscited by 2,342
  10. 2023PaLM-E: An Embodied Multimodal Language Modelcited by 2,071
  11. 2019Common Voice: A Massively-Multilingual Speech Corpuscited by 1,384
  12. 2023HuggingGPT: Solving AI Tasks with ChatGPT and its Friends in Hugging Facecited by 1,190
13 more
  1. 2021A General Language Assistant as a Laboratory for Alignmentcited by 1,055
  2. 2023LLaMA-Adapter: Efficient Fine-tuning of Language Models with Zero-init Attentioncited by 781
  3. 2023Neural Codec Language Models are Zero-Shot Text to Speech Synthesizerscited by 752
  4. 2022AudioLM: a Language Modeling Approach to Audio Generationcited by 616
  5. 2023Language Is Not All You Need: Aligning Perception with Language Modelscited by 543
  6. 2019Libri-Light: A Benchmark for ASR with Limited or No Supervisioncited by 467
  7. 2021GigaSpeech: An Evolving, Multi-domain ASR Corpus with 10,000 Hours of Transcribed Audiocited by 399
  8. 2021Generative Spoken Language Modeling from Raw Audiocited by 340
  9. 2023AudioGPT: Understanding and Generating Speech, Music, Sound, and Talking Headcited by 265
  10. 2021Speech Resynthesis from Discrete Disentangled Self-Supervised Representationscited by 243
  11. 2023Speak Foreign Languages with Your Own Voice: Cross-Lingual Neural Codec Language Modelingcited by 167
  12. 2023X-LLM: Bootstrapping Advanced Large Language Models by Treating Multi-Modalities as Foreign Languagescited by 128
  13. 2022Generative Spoken Dialogue Language Modelingcited by 126

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 54; 2024: 147; 2025: 183; 2026: 145.

529 papers.

Sort:

By area:

  1. 2023Video-LLaMA: An Instruction-tuned Audio-Visual Language Model for Video Understandingcited by 1,471
  2. 2023A Survey on Multimodal Large Language Modelscited by 909
  3. 2023NExT-GPT: Any-to-Any Multimodal LLMcited by 615
  4. 2023Qwen-Audio: Advancing Universal Audio Understanding via Unified Large-Scale Audio-Language Modelscited by 608
  5. 2024Qwen2-Audio Technical Reportcited by 560
  6. 2023SALMONN: Towards Generic Hearing Abilities for Large Language Modelscited by 506
  7. 2023AudioLDM 2: Learning Holistic Audio Generation with Self-supervised Pretrainingcited by 344
  8. 2023PandaGPT: One Model To Instruction-Follow Them Allcited by 334
  9. 2024CosyVoice 2: Scalable Streaming Speech Synthesis with Large Language Modelscited by 288
  10. 2024MM-LLMs: Recent Advances in MultiModal Large Language Modelscited by 278
  11. 2023LLaVAR: Enhanced Visual Instruction Tuning for Text-Rich Image Understandingcited by 274
  12. 2023Multimodal Foundation Models: From Specialists to General-Purpose Assistantscited by 247
38 more
  1. 2024AIR-Bench: Benchmarking Large Audio-Language Models via Generative Comprehensioncited by 220
  2. 2024AnyGPT: Unified Multimodal LLM with Discrete Sequence Modelingcited by 217
  3. 2024Mini-Omni: Language Models Can Hear, Talk While Thinking in Streamingcited by 210
  4. 2024VoiceBench: Benchmarking LLM-Based Voice Assistantscited by 200
  5. 2025VITA-1.5: Towards GPT-4o Level Real-Time Vision and Speech Interactioncited by 197
  6. 2023OneLLM: One Framework to Align All Modalities with Languagecited by 187
  7. 2025Multimodal Chain-of-Thought Reasoning: A Comprehensive Surveycited by 182
  8. 2024LLaMA-Omni: Seamless Speech Interaction with Large Language Modelscited by 173
  9. 2023SVIT: Scaling up Visual Instruction Tuningcited by 162
  10. 2023On decoder-only architecture for speech-to-text and large language model integrationcited by 145
  11. 2024Freeze-Omni: A Smart and Low Latency Speech-to-speech Dialogue Model with Frozen LLMcited by 141
  12. 2024WavTokenizer: an Efficient Acoustic Discrete Codec Tokenizer for Audio Language Modelingcited by 129
  13. 2024AudioBench: A Universal Benchmark for Audio Large Language Modelscited by 128
  14. 2024Spirit LM: Interleaved Spoken and Written Language Modelcited by 126
  15. 2025Advances and Challenges in Foundation Agents: From Brain-Inspired Intelligence to Evolutionary, Collaborative, and Safe Systemscited by 126
  16. 2025On The Landscape of Spoken Language Models: A Comprehensive Surveycited by 118
  17. 2024Large Language Models for Time Series: A Surveycited by 114
  18. 2023The Impact of Large Language Models on Scientific Discovery: a Preliminary Study using GPT-4cited by 112
  19. 2024WavLLM: Towards Robust and Adaptive Speech Large Language Modelcited by 109
  20. 2025Step-Audio 2 Technical Reportcited by 105
  21. 2023Toward General-Purpose Robots via Foundation Models: A Survey and Meta-Analysiscited by 103
  22. 2024video-SALMONN: Speech-Enhanced Audio-Visual Large Language Modelscited by 102
  23. 2024GroundingGPT:Language Enhanced Multi-modal Grounding Modelcited by 101
  24. 2023Joint Audio and Speech Understandingcited by 100
  25. 2025Audio-Reasoner: Improving Reasoning Capability in Large Audio Language Modelscited by 95
  26. 2023Spoken Question Answering and Speech Continuation Using Spectrogram-Powered LLMcited by 94
  27. 2024WavChat: A Survey of Spoken Dialogue Modelscited by 90
  28. 2025Perception, Reason, Think, and Plan: A Survey on Large Multimodal Reasoning Modelscited by 90
  29. 2024An Embarrassingly Simple Approach for LLM with Strong ASR Capacitycited by 88
  30. 2023VioLA: Unified Codec Language Models for Speech Recognition, Synthesis, and Translationcited by 87
  31. 2023SpeechTokenizer: Unified Speech Tokenizer for Speech Large Language Modelscited by 87
  32. 2024Recent Advances in Speech Language Models: A Surveycited by 87
  33. 2025Ola: Pushing the Frontiers of Omni-Modal Language Modelcited by 83
  34. 2024GPTVQ: The Blessing of Dimensionality for LLM Quantizationcited by 82
  35. 2023Connecting Speech Encoder and Large Language Model for ASRcited by 80
  36. 2024BadChain: Backdoor Chain-of-Thought Prompting for Large Language Modelscited by 80
  37. 2024Addressing Representation Collapse in Vector Quantized Models with One Linear Layercited by 80
  38. 2023BLSP: Bootstrapping Language-Speech Pre-training via Behavior Alignment of Continuation Writingcited by 79

50 most cited of 529 All 529 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2023Summary of ChatGPT-Related Research and Perspective Towards the Future of Large Language Models
  2. 2024DeSTA2: Developing Instruction-Following Speech Language Model Without Speech Instruction-Tuning Data
  3. 2026MoST: Mixing Speech and Text with Modality-Aware Mixture of Experts
  4. 2023ChatBridge: Bridging Modalities with Large Language Model as a Language Catalyst
  5. 2025DeepOmni: Towards Seamless and Smart Speech Interaction with Adaptive Modality-Specific MoE
  6. 2025Cross-Modal Knowledge Distillation for Speech Large Language Models
  7. 2023SwitchGPT: Adapting Large Language Models for Non-Text Outputs
  8. 2026X-OPD: Cross-Modal On-Policy Distillation for Capability Alignment in Speech LLMs