mlsanityData to 7 Oct 2026

AudioGPT: Understanding and Generating Speech, Music, Sound, and Talking Head

arXiv 2304.12995 ↗First posted 25 Apr 2023cs.CL cs.AI cs.SDcited by 264

Not on the map. Why?

Cited by 32 in Multimodal LLMs / vision-language models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Large language models (LLMs) have exhibited remarkable capabilities across a variety of domains and tasks, challenging our understanding of learning and cognition. Despite the recent success, current LLMs are not capable of processing complex audio information or conducting spoken conversations (like Siri or Alexa). In this work, we propose a multi-modal AI system named AudioGPT, which complements LLMs (i.e., ChatGPT) with 1) foundation models to process complex audio information and solve numerous understanding and generation tasks; and 2) the input/output interface (ASR, TTS) to support spoken dialogue. With an increasing demand to evaluate multi-modal LLMs of human intention understanding and cooperation with foundation models, we outline the principles and processes and test AudioGPT in terms of consistency, capability, and robustness. Experimental results demonstrate the capabilities of AudioGPT in solving AI tasks with speech, music, sound, and talking head understanding and generation in multi-round dialogues, which empower humans to create rich and diverse audio content with unprecedented ease. Our system is publicly available at urlhttps://github.com/AIGC-Audio/AudioGPT.

Key facts. Cited by 264 papers in the map (data through 7 Oct 2026). Builds on 43 of its 48 references that are in the map. First posted on arXiv 25 Apr 2023.

Builds on

43 of 48 references.

References · 43

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2022Training language models to follow instructions with human feedbackcited by 15,712
  4. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  5. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  6. 2020Scaling Laws for Neural Language Modelscited by 6,411
  7. 2022Robust Speech Recognition via Large-Scale Weak Supervisioncited by 3,972
  8. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,747
  9. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,464
  10. 2021HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Unitscited by 2,301
  11. 2022LaMDA: Language Models for Dialog Applicationscited by 1,218
  12. 2023HuggingGPT: Solving AI Tasks with ChatGPT and its Friends in Hugging Facecited by 1,173
31 more
  1. 2021SoundStream: An End-to-End Neural Audio Codeccited by 756
  2. 2020FastSpeech 2: Fast and High-Quality End-to-End Text to Speechcited by 710
  3. 2022High Fidelity Neural Audio Compressioncited by 673
  4. 2023Visual ChatGPT: Talking, Drawing and Editing with Visual Foundation Modelscited by 659
  5. 2022AudioLM: a Language Modeling Approach to Audio Generationcited by 611
  6. 2023Language Is Not All You Need: Aligning Perception with Language Modelscited by 543
  7. 2023MusicLM: Generating Music From Textcited by 442
  8. 2018Conv-TasNet: Surpassing Ideal Time-Frequency Magnitude Masking for Speech Separationcited by 400
  9. 2020A Simple Language Model for Task-Oriented Dialoguecited by 331
  10. 2023Make-An-Audio: Text-To-Audio Generation with Prompt-Enhanced Diffusion Modelscited by 323
  11. 2022Diffsound: Discrete Diffusion Model for Text-to-sound Generationcited by 256
  12. 2021DiffSinger: Singing Voice Synthesis via Shallow Diffusion Mechanismcited by 154
  13. 2022FastDiff: A Fast Conditional Diffusion Model for High-Quality Speech Synthesiscited by 136
  14. 2023GeneFace: Generalized and High-Fidelity Audio-Driven 3D Talking Face Synthesiscited by 131
  15. 2022Generative Spoken Dialogue Language Modelingcited by 125
  16. 2021Multi-Singer: Fast Multi-Singer Singing Voice Vocoder With A Large-Scale Corpuscited by 60
  17. 2020Multi-task Learning based Pre-trained Language Model for Code Completioncited by 58
  18. 2021Imitating Arbitrary Talking Style for Realistic Audio-DrivenTalking Face Synthesiscited by 56
  19. 2019Pythia: AI-assisted Code Completion Systemcited by 54
  20. 2022TF-GridNet: Making Time-Frequency Domain Models Great Again for Monaural Speaker Separationcited by 48
  21. 2022Separate What You Describe: Language-Queried Audio Source Separationcited by 45
  22. 2021VISinger: Variational Inference with Adversarial Learning for End-to-End Singing Voice Synthesiscited by 38
  23. 2020Language Model Prior for Low-Resource Neural Machine Translationcited by 37
  24. 2022TranSpeech: Speech-to-Speech Translation With Bilateral Perturbationcited by 34
  25. 2022textless-lib: a Library for Textless Spoken Language Processingcited by 28
  26. 2021Searchable Hidden Intermediates for End-to-End Models of Decomposable Sequence Taskscited by 24
  27. 2020DiscreTalk: Text-to-Speech as a Machine Translation Problemcited by 21
  28. 2022GenerSpeech: Towards Style Transfer for Generalizable Out-Of-Domain Text-to-Speechcited by 21
  29. 2018NeuralWarp: Time-Series Similarity with Warping Networkscited by 13
  30. 2021Improving the Performance of Automated Audio Captioning via Integrating the Acoustic and Semantic Informationcited by 11
  31. 2021Detect what you want: Target Sound Detectioncited by 4

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 55; 2024: 65; 2025: 93; 2026: 51.

264 papers.

Sort:

By area:

  1. 2023Video-LLaMA: An Instruction-tuned Audio-Visual Language Model for Video Understandingcited by 1,461
  2. 2023The Rise and Potential of Large Language Model Based Agents: A Surveycited by 1,422
  3. 2023NExT-GPT: Any-to-Any Multimodal LLMcited by 610
  4. 2023Qwen-Audio: Advancing Universal Audio Understanding via Unified Large-Scale Audio-Language Modelscited by 591
  5. 2023SpeechGPT: Empowering Large Language Models with Intrinsic Cross-Modal Conversational Abilitiescited by 518
  6. 2023PointLLM: Empowering Large Language Models to Understand Point Cloudscited by 326
  7. 2023Unified-IO 2: Scaling Autoregressive Multimodal Models with Vision, Language, Audio, and Actioncited by 301
  8. 2023MINT: Evaluating LLMs in Multi-turn Interaction with Tools and Language Feedbackcited by 293
  9. 2024MM-LLMs: Recent Advances in MultiModal Large Language Modelscited by 277
  10. 2023LLaVAR: Enhanced Visual Instruction Tuning for Text-Rich Image Understandingcited by 272
  11. 2023Multimodal Foundation Models: From Specialists to General-Purpose Assistantscited by 247
  12. 2023LLM Self Defense: By Self Examination, LLMs Know They Are Being Trickedcited by 226
38 more
  1. 2024AIR-Bench: Benchmarking Large Audio-Language Models via Generative Comprehensioncited by 215
  2. 2025Spatial-MLLM: Boosting MLLM Capabilities in Visual-based Spatial Intelligencecited by 206
  3. 2023UniControl: A Unified Diffusion Model for Controllable Visual Generation In the Wildcited by 197
  4. 2024Audio Flamingo: A Novel Audio Language Model with Few-Shot Learning and Dialogue Abilitiescited by 183
  5. 2024GAMA: A Large Audio-Language Model with Advanced Audio Understanding and Complex Reasoning Abilitiescited by 151
  6. 2023On decoder-only architecture for speech-to-text and large language model integrationcited by 144
  7. 2024AudioBench: A Universal Benchmark for Audio Large Language Modelscited by 124
  8. 2023The Impact of Large Language Models on Scientific Discovery: a Preliminary Study using GPT-4cited by 112
  9. 2023GraphLLM: Boosting Graph Reasoning Ability of Large Language Modelcited by 105
  10. 2025MMSU: A Massive Multi-task Spoken Language Understanding and Reasoning Benchmarkcited by 101
  11. 2025Step-Audio 2 Technical Reportcited by 101
  12. 2023Dynamic-SUPERB: Towards A Dynamic, Collaborative, and Comprehensive Instruction-Tuning Benchmark for Speechcited by 99
  13. 2023Joint Audio and Speech Understandingcited by 99
  14. 2024Large Multimodal Agents: A Surveycited by 99
  15. 2024Real3D-Portrait: One-shot Realistic 3D Talking Portrait Synthesiscited by 97
  16. 2024An Image Grid Can Be Worth a Video: Zero-shot Video Question Answering Using a VLMcited by 97
  17. 2023Spoken Question Answering and Speech Continuation Using Spectrogram-Powered LLMcited by 92
  18. 2023Make-An-Audio 2: Temporal-Enhanced Text-to-Audio Generationcited by 90
  19. 2024Recent Advances in Speech Language Models: A Surveycited by 85
  20. 2025Step-Audio: Unified Understanding and Generation in Intelligent Speech Interactioncited by 83
  21. 2023Connecting Speech Encoder and Large Language Model for ASRcited by 79
  22. 2025Ola: Pushing the Frontiers of Omni-Modal Language Modelcited by 79
  23. 2023LauraGPT: Listen, Attend, Understand, and Regenerate Audio with GPTcited by 77
  24. 2023Data-Copilot: Bridging Billions of Data and Humans with Autonomous Workflowcited by 76
  25. 2023Audio-Visual LLM for Video Understandingcited by 76
  26. 2023BLSP: Bootstrapping Language-Speech Pre-training via Behavior Alignment of Continuation Writingcited by 75
  27. 2023SALM: Speech-augmented Language Model with In-context Learning for Speech Recognition and Translationcited by 75
  28. 2024Detecting Multimedia Generated by Large AI Models: A Surveycited by 73
  29. 2023ChatPose: Chatting about 3D Human Posecited by 71
  30. 2023GPT4Point: A Unified Framework for Point-Language Understanding and Generationcited by 69
  31. 2024AppAgent v2: Advanced Agent for Flexible Mobile Interactionscited by 67
  32. 2023What Matters in Training a GPT4-Style Language Model with Multimodal Inputs?cited by 64
  33. 2024SpeechVerse: A Large-scale Generalizable Audio Language Modelcited by 62
  34. 2023TableGPT: Towards Unifying Tables, Nature Language and Commands into One GPTcited by 57
  35. 2024GraphInstruct: Empowering Large Language Models with Graph Understanding and Reasoning Capabilitycited by 57
  36. 2024A Comprehensive Review of Multimodal Large Language Models: Performance and Challenges Across Different Taskscited by 57
  37. 2023LLaSM: Large Language and Speech Modelcited by 56
  38. 2024A Comprehensive Survey of Foundation Models in Medicinecited by 56

50 most cited of 264 All 264 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024MMAU: A Massive Multi-Task Audio Understanding and Reasoning Benchmark
  2. 2025Audio Flamingo 2: An Audio-Language Model with Long-Audio Understanding and Expert Reasoning Abilities
  3. 2026Echo: Towards Advanced Audio Comprehension via Audio-Interleaved Reasoning
  4. 2025MMAR: A Challenging Benchmark for Deep Reasoning in Speech, Audio, Music, and Their Mix
  5. 2025Audio-Maestro: Enhancing Large Audio-Language Models with Tool-Augmented Reasoning
  6. 2025Audio MultiChallenge: A Multi-Turn Evaluation of Spoken Dialogue Systems on Natural Human Interaction
  7. 2025AU-Harness: An Open-Source Toolkit for Holistic Evaluation of Audio LLMs
  8. 2023Audiobox: Unified Audio Generation with Natural Language Prompts