mlsanityData to 7 Oct 2026

FunAudioLLM: Voice Understanding and Generation Foundation Models for Natural Interaction Between Humans and LLMs

arXiv 2407.04051 ↗First posted 4 Jul 2024cs.SD cs.AI eess.AScited by 118

Not on the map. Why?

Abstract (arXiv)

This report introduces FunAudioLLM, a model family designed to enhance natural voice interactions between humans and large language models (LLMs). At its core are two innovative models: SenseVoice, which handles multilingual speech recognition, emotion recognition, and audio event detection; and CosyVoice, which facilitates natural speech generation with control over multiple languages, timbre, speaking style, and speaker identity. SenseVoice-Small delivers exceptionally low-latency ASR for 5 languages, and SenseVoice-Large supports high-precision ASR for over 50 languages, while CosyVoice excels in multi-lingual voice generation, zero-shot in-context learning, cross-lingual voice cloning, and instruction-following capabilities. The models related to SenseVoice and CosyVoice have been open-sourced on Modelscope and Huggingface, along with the corresponding training, inference, and fine-tuning codes released on GitHub. By integrating these models with LLMs, FunAudioLLM enables applications such as speech-to-speech translation, emotional voice chat, interactive podcasts, and expressive audiobook narration, thereby pushing the boundaries of voice interaction technology. Demos are available at https://fun-audio-llm.github.io, and the code can be accessed at https://github.com/FunAudioLLM.

Key facts. Cited by 118 papers in the map (data through 7 Oct 2026). Builds on 45 of its 61 references that are in the map. First posted on arXiv 4 Jul 2024.

Builds on

45 of 61 references.

References · 45

  1. 2017Attention Is All You Needcited by 59,932
  2. 2023GPT-4 Technical Reportcited by 18,132
  3. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  4. 2022Classifier-Free Diffusion Guidancecited by 5,717
  5. 2022Flow Matching for Generative Modelingcited by 5,426
  6. 2022Robust Speech Recognition via Large-Scale Weak Supervisioncited by 3,972
  7. 2023Qwen Technical Reportcited by 3,273
  8. 2024Gemini 1.5: Unlocking multimodal understanding across millions of tokens of contextcited by 3,175
  9. 2021HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Unitscited by 2,301
  10. 2019Common Voice: A Massively-Multilingual Speech Corpuscited by 1,358
  11. 2021SoundStream: An End-to-End Neural Audio Codeccited by 756
  12. 2023Neural Codec Language Models are Zero-Shot Text to Speech Synthesizerscited by 737
33 more
  1. 2022High Fidelity Neural Audio Compressioncited by 673
  2. 2019LibriTTS: A Corpus Derived from LibriSpeech for Text-to-Speechcited by 639
  3. 2018MELD: A Multimodal Multi-Party Dataset for Emotion Recognition in Conversationscited by 636
  4. 2023Qwen-Audio: Advancing Universal Audio Understanding via Unified Large-Scale Audio-Language Modelscited by 591
  5. 2019PANNs: Large-Scale Pretrained Audio Neural Networks for Audio Pattern Recognitioncited by 563
  6. 2023SALMONN: Towards Generic Hearing Abilities for Large Language Modelscited by 492
  7. 2022BEATs: Audio Pre-Training with Acoustic Tokenizerscited by 378
  8. 2017AISHELL-1: An Open-Source Mandarin Speech Corpus and A Speech Recognition Baselinecited by 364
  9. 2023emotion2vec: Self-Supervised Pre-Training for Speech Emotion Representationcited by 196
  10. 2021WenetSpeech: A 10000+ Hours Multi-domain Mandarin Corpus for Speech Recognitioncited by 189
  11. 2023Speak Foreign Languages with Your Own Voice: Cross-Lingual Neural Codec Language Modelingcited by 166
  12. 2023Matcha-TTS: A fast TTS architecture with conditional flow matchingcited by 142
  13. 2022Paraformer: Fast and Accurate Parallel Transformer for Non-autoregressive End-to-End Speech Recognitioncited by 139
  14. 2018AISHELL-2: Transforming Mandarin ASR Research Into Industrial Scalecited by 125
  15. 2023CAM++: A Fast and Efficient Network for Speaker Verification Using Context-Aware Maskingcited by 95
  16. 2023HiFi-Codec: Group-residual Vector quantization for High Fidelity Audio Codeccited by 95
  17. 2023FunASR: A Fundamental End-to-End Speech Recognition Toolkitcited by 92
  18. 2018End-to-End Automatic Speech Translation of Audiobookscited by 65
  19. 2020Coswara -- A Database of Breathing, Cough, and Voice Sounds for COVID-19 Diagnosiscited by 58
  20. 2021Sound Event Detection: A Tutorialcited by 51
  21. 2023TextrolSpeech: A Text Style Control Speech Corpus With Codec Language Text-to-Speech Modelscited by 48
  22. 2023FunCodec: A Fundamental, Reproducible and Integrable Open-source Toolkit for Neural Speech Codeccited by 44
  23. 2023An Enhanced Res2Net with Local and Global Feature Fusion for Speaker Verificationcited by 41
  24. 2024EmoBox: Multilingual Multi-corpus Speech Emotion Recognition Toolkit and Benchmarkcited by 39
  25. 2023PromptTTS++: Controlling Speaker Identity in Prompt-Based Text-to-Speech Using Natural Language Descriptionscited by 25
  26. 2020Controllable Time-Delay Transformer for Real-Time Punctuation Prediction and Disfluency Detectioncited by 23
  27. 2023E-chat: Emotion-sensitive Spoken Dialogue System with Large Language Modelscited by 19
  28. 2020SAN-M: Memory Equipped Self-Attention for End-to-End Speech Recognitioncited by 16
  29. 2023SeACo-Paraformer: A Non-Autoregressive ASR System with Flexible and Effective Hotword Customization Abilitycited by 16
  30. 2024Making Flow-Matching-Based Zero-Shot Text-to-Speech Laugh as You Likecited by 11
  31. 2023HiFTNet: A Fast High-Quality Neural Vocoder with Harmonic-plus-Noise Filter and Inverse Short Time Fourier Transformcited by 10
  32. 2022NewsPod: Automatic and Interactive News Podcastscited by 6
  33. 2022Text-driven Emotional Style Control and Cross-speaker Style Transfer in Neural TTScited by 6

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 14; 2025: 59; 2026: 45.

118 papers.

Sort:

By area:

  1. 2025CosyVoice 3: Towards In-the-wild Speech Generation via Scaling-up and Post-trainingcited by 142
  2. 2024WavTokenizer: an Efficient Acoustic Discrete Codec Tokenizer for Audio Language Modelingcited by 124
  3. 2025On The Landscape of Spoken Language Models: A Comprehensive Surveycited by 116
  4. 2025A Comprehensive Survey of Mixture-of-Experts: Algorithms, Theory, and Applicationscited by 91
  5. 2024WavChat: A Survey of Spoken Dialogue Modelscited by 88
  6. 2024Recent Advances in Speech Language Models: A Surveycited by 85
  7. 2025LLaMA-Omni2: LLM-based Real-time Spoken Chatbot with Autoregressive Streaming Speech Synthesiscited by 75
  8. 2025HumanOmni: A Large Vision-Speech Language Model for Human-Centric Video Understandingcited by 58
  9. 2024OmniFlatten: An End-to-end GPT Model for Seamless Voice Conversationcited by 57
  10. 2024Baichuan-Omni Technical Reportcited by 42
  11. 2024Zero-shot Voice Conversion with Diffusion Transformerscited by 39
  12. 2024AdvWave: Stealthy Adversarial Jailbreak Attack against Large Audio-Language Modelscited by 33
38 more
  1. 2025VITA-Audio: Fast Interleaved Cross-Modal Token Generation for Efficient Large Speech-Language Modelcited by 30
  2. 2025F5R-TTS: Improving Flow-Matching based Text-to-Speech with Group Relative Policy Optimizationcited by 24
  3. 2026OmniVoice: Towards Omnilingual Zero-Shot Text-to-Speech with Diffusion Language Modelscited by 22
  4. 2025Fun-ASR Technical Reportcited by 19
  5. 2025WavRAG: Audio-Integrated Retrieval Augmented Generation for Spoken Dialogue Modelscited by 18
  6. 2025Jailbreak-AudioBench: In-Depth Evaluation and Analysis of Jailbreak Threats for Large Audio Language Modelscited by 17
  7. 2025OmniAudio: Generating Spatial Audio from 360-Degree Videocited by 17
  8. 2025S2S-Arena: Evaluating Paralinguistic Instruction Following in Speech-to-Speech Modelscited by 16
  9. 2025VocalNet: Speech LLM with Multi-Token Prediction for Faster and High-Quality Generationcited by 16
  10. 2025AudioJailbreak: Jailbreak Attacks against End-to-End Large Audio-Language Modelscited by 16
  11. 2025WavReward: Spoken Dialogue Models With Generalist Reward Evaluatorscited by 15
  12. 2025Audio Jailbreak: An Open Comprehensive Benchmark for Jailbreaking Large Audio-Language Modelscited by 15
  13. 2025Differentiable Reward Optimization for LLM based TTS systemcited by 15
  14. 2025ContextASR-Bench: A Massive Contextual Speech Recognition Benchmarkcited by 14
  15. 2025NVSpeech: An Integrated and Scalable Pipeline for Human-Like Speech Modeling with Paralinguistic Vocalizationscited by 14
  16. 2025OSUM: Advancing Open Speech Understanding Models with Limited Resources in Academiacited by 13
  17. 2026X-OPD: Cross-Modal On-Policy Distillation for Capability Alignment in Speech LLMscited by 13
  18. 2025TeleAntiFraud-28k: An Audio-Text Slow-Thinking Dataset for Telecom Fraud Detectioncited by 12
  19. 2025Towards Spoken Mathematical Reasoning: Benchmarking Speech-based Models over Multi-faceted Math Problemscited by 12
  20. 2025SwitchLingua: The First Large-Scale Multilingual and Multi-Ethnic Code-Switching Datasetcited by 12
  21. 2024Enhancing Low-Resource ASR through Versatile TTS: Bridging the Data Gapcited by 11
  22. 2025CS-Dialogue: A 104-Hour Dataset of Spontaneous Mandarin-English Code-Switching Dialogues for Speech Recognitioncited by 10
  23. 2025Pseudo-Autoregressive Neural Codec Language Models for Efficient Zero-Shot Text-to-Speech Synthesiscited by 10
  24. 2026Audio-Omni: Extending Multi-modal Understanding to Versatile Audio Generation and Editingcited by 10
  25. 2024Boosting Code-Switching ASR with Mixture of Experts Enhanced Speech-Conditioned LLMcited by 9
  26. 2025OmniChat: Enhancing Spoken Dialogue Systems with Scalable Synthetic Data for Diverse Scenarioscited by 9
  27. 2026Omni-Diffusion: Unified Multimodal Understanding and Generation with Masked Discrete Diffusioncited by 9
  28. 2024MERaLiON-AudioLLM: Bridging Audio and Language with Large Language Modelscited by 7
  29. 2025A Self-Refining Framework for Enhancing ASR Using TTS-Synthesized Datacited by 7
  30. 2025WenetSpeech-Chuan: A Large-Scale Sichuanese Corpus with Rich Annotation for Dialectal Speech Processingcited by 7
  31. 2025`From Prompt to Perturbation': An Adaptive Framework for Voice-Based Jailbreaks on Audio LLMscited by 6
  32. 2025DrVoice: Parallel Speech-Text Voice Conversation Model via Dual-Resolution Speech Representationscited by 6
  33. 2025SynParaSpeech: Automated Synthesis of Paralinguistic Datasets for Speech Generation and Understandingcited by 6
  34. 2025MNV-17: A High-Quality Performative Mandarin Dataset for Nonverbal Vocalization Recognition in Speechcited by 6
  35. 2025Steering Language Model to Stable Speech Emotion Recognition via Contextual Perception and Chain of Thoughtcited by 5
  36. 2024LA-RAG:Enhancing LLM-based ASR Accuracy with Retrieval-Augmented Generationcited by 4
  37. 2025ChipChat: Low-Latency Cascaded Conversational Agent in MLXcited by 4
  38. 2025HiKE: Hierarchical Evaluation Framework for Korean-English Code-Switching Speech Recognitioncited by 4

50 most cited of 118 All 118 (JSON)