mlsanityData to 14 Oct 2026

UniAudio: An Audio Foundation Model Toward Universal Audio Generation

arXiv 2310.00704 ↗First posted 1 Oct 2023cs.SD eess.AScited by 127

Not on the map. Why?

Cited by 20 in Diffusion models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Large Language models (LLM) have demonstrated the capability to handle a variety of generative tasks. This paper presents the UniAudio system, which, unlike prior task-specific approaches, leverages LLM techniques to generate multiple types of audio (including speech, sounds, music, and singing) with given input conditions. UniAudio 1) first tokenizes all types of target audio along with other condition modalities, 2) concatenates source-target pair as a single sequence, and 3) performs next-token prediction using LLM. Also, a multi-scale Transformer model is proposed to handle the overly long sequences caused by the residual vector quantization based neural codec in tokenization. Training of UniAudio is scaled up to 165K hours of audio and 1B parameters, based on all generative tasks, aiming to obtain sufficient prior knowledge not only in the intrinsic properties of audio but also the inter-relationship between audio and other modalities. Therefore, the trained UniAudio model has the potential to become a foundation model for universal audio generation: it shows strong capability in all trained tasks and can seamlessly support new audio generation tasks after simple fine-tuning. Experiments demonstrate that UniAudio achieves state-of-the-art or at least competitive results on most of the 11 tasks. Demo and code are released at https://github.com/yangdongchao/UniAudio

Key facts. Cited by 127 papers in the map (data through 14 Oct 2026). First posted on arXiv 1 Oct 2023.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 3; 2024: 60; 2025: 47; 2026: 17.

127 papers.

Sort:

By area:

  1. 2024Moshi: a speech-text foundation model for real-time dialoguecited by 568
  2. 2024CosyVoice: A Scalable Multilingual Zero-shot Text-to-speech Synthesizer based on Supervised Semantic Tokenscited by 334
  3. 2024NaturalSpeech 3: Zero-Shot Speech Synthesis with Factorized Codec and Diffusion Modelscited by 226
  4. 2025Kimi-Audio Technical Reportcited by 222
  5. 2024Fast Timing-Conditioned Latent Audio Diffusioncited by 181
  6. 2024VALL-E 2: Neural Codec Language Models are Human Parity Zero-Shot Text to Speech Synthesizerscited by 122
  7. 2023Audiobox: Unified Audio Generation with Natural Language Promptscited by 120
  8. 2026Qwen3-TTS Technical Reportcited by 118
  9. 2024Large Language Models for Time Series: A Surveycited by 114
  10. 2024WavChat: A Survey of Spoken Dialogue Modelscited by 90
  11. 2024Recent Advances in Speech Language Models: A Surveycited by 87
  12. 2023LauraGPT: Listen, Attend, Understand, and Regenerate Audio with GPTcited by 78
38 more
  1. 2024Codec Does Matter: Exploring the Semantic Shortcoming of Codec for Audio Language Modelcited by 72
  2. 2024Foundation Models for Music: A Surveycited by 37
  3. 2023HierSpeech++: Bridging the Gap between Semantic and Acoustic Representation of Speech by Hierarchical Variational Inference for Zero-shot Speech Synthesiscited by 35
  4. 2024LLMs Meet Multimodal Generation and Editing: A Surveycited by 35
  5. 2024Autoregressive Diffusion Transformer for Text-to-Speech Synthesiscited by 34
  6. 2024The Codecfake Dataset and Countermeasures for the Universally Detection of Deepfake Audiocited by 31
  7. 2024Taming Data and Transformers for Audio Generationcited by 28
  8. 2024Next Token Prediction Towards Multimodal Intelligence: A Comprehensive Surveycited by 27
  9. 2024RALL-E: Robust Codec Language Modeling with Chain-of-Thought Prompting for Text-to-Speech Synthesiscited by 26
  10. 2024Both Ears Wide Open: Towards Language-Driven Spatial Audio Generationcited by 25
  11. 2024FlashSpeech: Efficient Zero-Shot Speech Synthesiscited by 24
  12. 2024MaskSR: Masked Language Model for Full-band Speech Restorationcited by 24
  13. 2024SyllableLM: Learning Coarse Semantic Units for Speech Language Modelscited by 24
  14. 2024SimpleSpeech: Towards Simple and Efficient Text-to-Speech with Scalar Latent Transformer Diffusion Modelscited by 23
  15. 2024Towards Controllable Speech Synthesis in the Era of Large Language Models: A Systematic Surveycited by 23
  16. 2024Beyond Language Models: Byte Models are Digital World Simulatorscited by 22
  17. 2024Improving Robustness of LLM-based Speech Synthesis by Learning Monotonic Alignmentcited by 22
  18. 2025FreeAudio: Training-Free Timing Planning for Controllable Long-Form Text-to-Audio Generationcited by 21
  19. 2025Scaling Laws of Motion Forecasting and Planning -- Technical Reportcited by 19
  20. 2024Prompt-guided Precise Audio Editing with Diffusion Modelscited by 18
  21. 2024SOLAMI: Social Vision-Language-Action Modeling for Immersive Interaction with 3D Autonomous Characterscited by 18
  22. 2024SimpleSpeech 2: Towards Simple and Efficient Text-to-Speech with Flow-based Scalar Latent Transformer Diffusion Modelscited by 17
  23. 2024Towards Rationality in Language and Multimodal Agents: A Surveycited by 16
  24. 2024Instruct-MusicGen: Unlocking Text-to-Music Editing for Music Language Models via Instruction Tuningcited by 15
  25. 2024MusicFlow: Cascaded Flow Matching for Text Guided Music Generationcited by 15
  26. 2025High-Fidelity Simultaneous Speech-To-Speech Translationcited by 15
  27. 2024Paralinguistics-Aware Speech-Empowered Large Language Models for Natural Conversationcited by 14
  28. 2024Text-to-Song: Towards Controllable Music Generation Incorporating Vocals and Accompanimentcited by 13
  29. 2024Prompt-Singer: Controllable Singing-Voice-Synthesis with Natural Language Promptcited by 12
  30. 2024PeriodWave: Multi-Period Flow Matching for High-Fidelity Waveform Generationcited by 12
  31. 2024SongCreator: Lyrics-based Universal Song Generationcited by 12
  32. 2025Survey on AI-Generated Media Detection: From Non-MLLM to MLLMcited by 12
  33. 2025Slamming: Training a Speech Language Model on One GPU in a Daycited by 12
  34. 2024Making Flow-Matching-Based Zero-Shot Text-to-Speech Laugh as You Likecited by 11
  35. 2025Do we really have to filter out random noise in pre-training data for language models?cited by 11
  36. 2025Aligning Spoken Dialogue Models from User Interactionscited by 11
  37. 2025FLM-Audio: Natural Monologues Improves Native Full-Duplex Chatbots via Dual Trainingcited by 11
  38. 2024A Survey on Multimodal Wearable Sensor-based Human Action Recognitioncited by 10

50 most cited of 127 All 127 (JSON)