mlsanityData to 7 Oct 2026

Baichuan-Audio: A Unified Framework for End-to-End Speech Interaction

arXiv 2502.17239 ↗First posted 24 Feb 2025cs.CL cs.SD eess.AScited by 68

Not on the map. Why?

Abstract (arXiv)

We introduce Baichuan-Audio, an end-to-end audio large language model that seamlessly integrates audio understanding and generation. It features a text-guided aligned speech generation mechanism, enabling real-time speech interaction with both comprehension and generation capabilities. Baichuan-Audio leverages a pre-trained ASR model, followed by multi-codebook discretization of speech at a frame rate of 12.5 Hz. This multi-codebook setup ensures that speech tokens retain both semantic and acoustic information. To further enhance modeling, an independent audio head is employed to process audio tokens, effectively capturing their unique characteristics. To mitigate the loss of intelligence during pre-training and preserve the original capabilities of the LLM, we propose a two-stage pre-training strategy that maintains language understanding while enhancing audio modeling. Following alignment, the model excels in real-time speech-based conversation and exhibits outstanding question-answering capabilities, demonstrating its versatility and efficiency. The proposed model demonstrates superior performance in real-time spoken dialogue and exhibits strong question-answering abilities. Our code, model and training data are available at https://github.com/baichuan-inc/Baichuan-Audio

Key facts. Cited by 68 papers in the map (data through 7 Oct 2026). Builds on 36 of its 44 references that are in the map. First posted on arXiv 24 Feb 2025.

Builds on

36 of 44 references.

References · 36

  1. 2022Flow Matching for Generative Modelingcited by 5,426
  2. 2022Robust Speech Recognition via Large-Scale Weak Supervisioncited by 3,972
  3. 2017TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehensioncited by 2,895
  4. 2021HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Unitscited by 2,301
  5. 2021WavLM: Large-Scale Self-Supervised Pre-Training for Full Stack Speech Processingcited by 1,609
  6. 2020HiFi-GAN: Generative Adversarial Networks for Efficient and High Fidelity Speech Synthesiscited by 1,139
  7. 2024MiniCPM-V: A GPT-4V Level MLLM on Your Phonecited by 1,061
  8. 2021SoundStream: An End-to-End Neural Audio Codeccited by 756
  9. 2023Neural Codec Language Models are Zero-Shot Text to Speech Synthesizerscited by 737
  10. 2022High Fidelity Neural Audio Compressioncited by 673
  11. 2023Qwen-Audio: Advancing Universal Audio Understanding via Unified Large-Scale Audio-Language Modelscited by 591
  12. 2024Qwen2-Audio Technical Reportcited by 543
24 more
  1. 2024Moshi: a speech-text foundation model for real-time dialoguecited by 536
  2. 2022FLEURS: Few-shot Learning Evaluation of Universal Representations of Speechcited by 458
  3. 2023CMMLU: Measuring massive multitask language understanding in Chinesecited by 436
  4. 2023High-Fidelity Audio Compression with Improved RVQGANcited by 423
  5. 2022UTMOS: UTokyo-SaruLab System for VoiceMOS Challenge 2022cited by 381
  6. 2024CosyVoice: A Scalable Multilingual Zero-shot Text-to-speech Synthesizer based on Supervised Semantic Tokenscited by 323
  7. 2024VILA-U: a Unified Foundation Model Integrating Visual Understanding and Generationcited by 290
  8. 2024GLM-4-Voice: Towards Intelligent and Human-Like End-to-End Spoken Chatbotcited by 218
  9. 2023Pengi: An Audio Language Model for Audio Taskscited by 217
  10. 2025VITA-1.5: Towards GPT-4o Level Real-Time Vision and Speech Interactioncited by 191
  11. 2021WenetSpeech: A 10000+ Hours Multi-domain Mandarin Corpus for Speech Recognitioncited by 189
  12. 2024LLaMA-Omni: Seamless Speech Interaction with Large Language Modelscited by 169
  13. 2023Matcha-TTS: A fast TTS architecture with conditional flow matchingcited by 142
  14. 2024Freeze-Omni: A Smart and Low Latency Speech-to-speech Dialogue Model with Frozen LLMcited by 135
  15. 2024Spirit LM: Interleaved Spoken and Written Language Modelcited by 123
  16. 2024WavLLM: Towards Robust and Adaptive Speech Large Language Modelcited by 107
  17. 2023Textually Pretrained Speech Language Modelscited by 99
  18. 2023Spoken Question Answering and Speech Continuation Using Spectrogram-Powered LLMcited by 92
  19. 2024Autoregressive Speech Synthesis without Vector Quantizationcited by 65
  20. 2024Scaling Speech-Text Pre-training with Synthetic Interleaved Datacited by 45
  21. 2024Baichuan-Omni Technical Reportcited by 42
  22. 2024SALMONN-omni: A Codec-free LLM for Full-duplex Speech Understanding and Generationcited by 33
  23. 2023A Chat About Boring Problems: Studying GPT-based text normalizationcited by 10
  24. 2024Why Do Speech Language Models Fail to Generate Semantically Coherent Outputs? A Modality Evolving Perspectivecited by 8

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 37; 2026: 31.

68 papers.

Sort:

By area:

  1. 2025Kimi-Audio Technical Reportcited by 210
  2. 2025MiMo-Audio: Audio Language Models are Few-Shot Learnerscited by 58
  3. 2025DualToken: Towards Unifying Visual Understanding and Generation with Dual Visual Vocabulariescited by 42
  4. 2025VITA-Audio: Fast Interleaved Cross-Modal Token Generation for Efficient Large Speech-Language Modelcited by 30
  5. 2026LongCat-Next: Lexicalizing Modalities as Discrete Tokenscited by 30
  6. 2025InteractiveOmni: A Unified Omni-modal Model for Audio-Visual Multi-turn Dialoguecited by 27
  7. 2025SALMONN-omni: A Standalone Speech LLM without Codec Injection for Full-duplex Conversationcited by 25
  8. 2025STITCH: Simultaneous Thinking and Talking with Chunked Reasoning for Spoken Language Modelscited by 25
  9. 2025LongCat-Flash-Omni Technical Reportcited by 23
  10. 2025A Survey of Token Compression for Efficient Multimodal Large Language Modelscited by 22
  11. 2025XY-Tokenizer: Mitigating the Semantic-Acoustic Conflict in Low-Bitrate Speech Codecscited by 21
  12. 2025MTR-DuplexBench: Towards a Comprehensive Evaluation of Multi-Round Conversations for Full-Duplex Speech Language Modelscited by 20
38 more
  1. 2025Fun-Audio-Chat Technical Reportcited by 17
  2. 2025SHANKS: Simultaneous Hearing and Thinking for Spoken Language Modelscited by 16
  3. 2025Ming-UniAudio: Speech LLM for Joint Understanding, Generation and Editing with Unified Representationcited by 15
  4. 2026Audio Flamingo Next: Next-Generation Open Audio-Language Models for Speech, Sound, and Musiccited by 15
  5. 2025ContextASR-Bench: A Massive Contextual Speech Recognition Benchmarkcited by 14
  6. 2025TaDiCodec: Text-aware Diffusion Speech Tokenizer for Speech Language Modelingcited by 12
  7. 2025WildSpeech-Bench: Benchmarking End-to-End SpeechLLMs in the Wildcited by 11
  8. 2026MoshiRAG: Asynchronous Knowledge Retrieval for Full-Duplex Speech Language Modelscited by 11
  9. 2026MOSS-TTS Technical Reportcited by 10
  10. 2025VStyle: A Benchmark for Voice Style Adaptation with Spoken Instructionscited by 9
  11. 2026MOSS-Audio Technical Reportcited by 9
  12. 2025SageLM: A Multi-aspect and Explainable Large Language Model for Speech Judgementcited by 7
  13. 2026Covo-Audio Technical Reportcited by 7
  14. 2025AURA: Agent for Understanding, Reasoning, and Automated Tool Use in Voice-Driven Taskscited by 5
  15. 2025Audio Flamingo Sound-CoT Technical Report: Improving Chain-of-Thought Reasoning in Sound Understandingcited by 5
  16. 2026The Silent Thought: Modeling Internal Cognition in Full-Duplex Spoken Dialogue Models via Latent Reasoningcited by 5
  17. 2025VCB Bench: An Evaluation Benchmark for Audio-Grounded Large Language Model Conversational Agentscited by 4
  18. 2025EchoingPixels: Aliasing-Resistant Joint Token Reduction for Audio-Visual LLMscited by 4
  19. 2026Do We Need Distinct Representations for Every Speech Token? Unveiling and Exploiting Redundancy in Large Speech Language Modelscited by 4
  20. 2026Raon-Speech Technical Reportcited by 4
  21. 2025S2SBench: A Benchmark for Quantifying Intelligence Degradation in Speech-to-Speech Large Language Modelscited by 3
  22. 2025From Text to Talk: Audio-Language Model Needs Non-Autoregressive Joint Trainingcited by 3
  23. 2025End-to-end Listen, Look, Speak and Actcited by 3
  24. 2025GOAT-SLM: A Spoken Language Model with Paralinguistic and Speaker Characteristic Awarenesscited by 2
  25. 2025VoxRole: A Comprehensive Benchmark for Evaluating Speech-Based Role-Playing Agentscited by 2
  26. 2025Extending Audio Context for Long-Form Understanding in Large Audio-Language Modelscited by 2
  27. 2025Data-Centric Lessons To Improve Speech-Language Pretrainingcited by 2
  28. 2025ALMGuard: Safety Shortcuts and Where to Find Them as Guardrails for Audio-Language Modelscited by 2
  29. 2026SonicBench: Dissecting the Physical Perception Bottleneck in Large Audio Language Modelscited by 2
  30. 2026DIFFA-2: A Practical Diffusion Large Language Model for General Audio Understandingcited by 2
  31. 2026Scaling Speech Tokenizers with Diffusion Autoencoderscited by 2
  32. 2026Audio-Visual Flamingo: Open Audio-Visual Intelligence for Long and Complex Videoscited by 2
  33. 2025Efficient Speech Language Modeling via Energy Distance in Continuous Latent Spacecited by 1
  34. 2025What Makes a Good Speech Tokenizer for LLM-Centric Speech Generation? A Systematic Studycited by 1
  35. 2025DeepOmni: Towards Seamless and Smart Speech Interaction with Adaptive Modality-Specific MoEcited by 1
  36. 2026Human or Machine? A Preliminary Turing Test for Speech-to-Speech Interactioncited by 1
  37. 2026TiCo: Time-Controllable Spoken Dialogue Modelcited by 1
  38. 2026VITA-QinYu: Expressive Spoken Language Model for Role-Playing and Singingcited by 1

50 most cited of 68 All 68 (JSON)