mlsanityData to 14 Oct 2026

SLAM-Omni: Timbre-Controllable Voice Interaction System with Single-Stage Training

arXiv 2412.15649 ↗First posted 20 Dec 2024eess.AScited by 59

Not on the map. Why?

Abstract (arXiv)

Recent advancements highlight the potential of end-to-end real-time spoken dialogue systems, showcasing their low latency and high quality. In this paper, we introduce SLAM-Omni, a timbre-controllable, end-to-end voice interaction system with single-stage training. SLAM-Omni achieves zero-shot timbre control by modeling spoken language with semantic tokens and decoupling speaker information to a vocoder. By predicting grouped speech semantic tokens at each step, our method significantly reduces the sequence length of audio tokens, accelerating both training and inference. Additionally, we propose historical text prompting to compress dialogue history, facilitating efficient multi-round interactions. Comprehensive evaluations reveal that SLAM-Omni outperforms prior models of similar scale, requiring only 15 hours of training on 4 GPUs with limited data. Notably, it is the first spoken dialogue system to achieve competitive performance with a single-stage training approach, eliminating the need for pre-training on TTS or ASR tasks. Further experiments validate its multilingual and multi-turn dialogue capabilities on larger datasets.

Key facts. Cited by 59 papers in the map (data through 14 Oct 2026). First posted on arXiv 20 Dec 2024.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 36; 2026: 23.

59 papers.

Sort:

By area:

  1. 2025On The Landscape of Spoken Language Models: A Comprehensive Surveycited by 118
  2. 2025LLaMA-Omni2: LLM-based Real-time Spoken Chatbot with Autoregressive Streaming Speech Synthesiscited by 76
  3. 2025Recent Advances in Discrete Speech Tokens: A Reviewcited by 43
  4. 2025EmoVoice: LLM-based Emotional Text-To-Speech Model with Freestyle Text Promptingcited by 34
  5. 2025VITA-Audio: Fast Interleaved Cross-Modal Token Generation for Efficient Large Speech-Language Modelcited by 31
  6. 2025URO-Bench: Towards Comprehensive Evaluation for End-to-End Spoken Dialogue Modelscited by 30
  7. 2025SALMONN-omni: A Standalone Speech LLM without Codec Injection for Full-duplex Conversationcited by 27
  8. 2025STITCH: Simultaneous Thinking and Talking with Chunked Reasoning for Spoken Language Modelscited by 26
  9. 2025Mini-Omni-Reasoner: Token-Level Thinking-in-Speaking in Large Speech Modelscited by 21
  10. 2025FD-Bench: A Full-Duplex Benchmarking Pipeline Designed for Full Duplex Spoken Dialogue Systemscited by 20
  11. 2025Fun-Audio-Chat Technical Reportcited by 19
  12. 2025VocalBench: Benchmarking the Vocal Conversational Abilities for Speech Interaction Modelscited by 17
38 more
  1. 2025VocalNet: Speech LLM with Multi-Token Prediction for Faster and High-Quality Generationcited by 16
  2. 2025WavReward: Spoken Dialogue Models With Generalist Reward Evaluatorscited by 15
  3. 2025EchoMind: An Interrelated Multi-level Benchmark for Evaluating Empathetic Speech Language Modelscited by 11
  4. 2025OpenS2S: Advancing Fully Open-Source End-to-End Empathetic Large Speech Language Modelcited by 9
  5. 2025PredGen: Accelerated Inference of Large Language Models through Input-Time Speculation for Real-Time Speech Interactioncited by 8
  6. 2025DIFFA: Large Language Diffusion Models Can Listen and Understandcited by 8
  7. 2026SLAM-LLM: A Modular, Open-Source Multimodal Large Language Model Framework and Best Practice for Speech, Language, Audio and Music Processingcited by 8
  8. 2025SageLM: A Multi-aspect and Explainable Large Language Model for Speech Judgementcited by 7
  9. 2025Evaluating Bias in Spoken Dialogue LLMs for Real-World Decisions and Recommendationscited by 7
  10. 2025DrVoice: Parallel Speech-Text Voice Conversation Model via Dual-Resolution Speech Representationscited by 6
  11. 2026WavBench: Benchmarking Reasoning, Colloquialism, and Paralinguistics for End-to-End Spoken Dialogue Modelscited by 6
  12. 2025Enhancing Speech-to-Speech Dialogue Modeling with End-to-End Retrieval-Augmented Generationcited by 5
  13. 2025CCFQA: A Benchmark for Cross-Lingual and Cross-Modal Speech and Text Factuality Evaluationcited by 5
  14. 2025End-to-end Listen, Look, Speak and Actcited by 5
  15. 2025Empathy Omni: Enabling Empathetic Speech Response Generation through Large Language Modelscited by 3
  16. 2025From Text to Talk: Audio-Language Model Needs Non-Autoregressive Joint Trainingcited by 3
  17. 2026DIFFA-2: A Practical Diffusion Large Language Model for General Audio Understandingcited by 3
  18. 2026Still Between Us? Evaluating and Improving Voice Assistant Robustness to Third-Party Interruptionscited by 3
  19. 2026Reinforcement Learning with Evolving Rubrics as Rewards for Audio Reasoningcited by 3
  20. 2025GOAT-SLM: A Spoken Language Model with Paralinguistic and Speaker Characteristic Awarenesscited by 2
  21. 2025CS3-Bench: Evaluating and Enhancing Speech-to-Speech LLMs for Mandarin-English Code-Switchingcited by 2
  22. 2025UltraVoice: Scaling Fine-Grained Style-Controlled Speech Conversations for Spoken Dialogue Modelscited by 2
  23. 2026Ex-Omni: Enabling 3D Facial Animation Generation for Omni-modal Large Language Modelscited by 2
  24. 2026Vox-Infinity: Benchmarking the Limits of Long-Context Spoken Language Modelscited by 2
  25. 2025Towards Efficient Speech-Text Jointly Decoding within One Speech Language Modelcited by 1
  26. 2025What Makes a Good Speech Tokenizer for LLM-Centric Speech Generation? A Systematic Studycited by 1
  27. 2025DeepOmni: Towards Seamless and Smart Speech Interaction with Adaptive Modality-Specific MoEcited by 1
  28. 2025WEST: LLM based Speech Toolkit for Speech Understanding, Generation, and Interactioncited by 1
  29. 2025UniVoice: Unifying Autoregressive ASR and Flow-Matching based TTS with Large Language Modelscited by 1
  30. 2026VocalNet-MDM: Accelerating Streaming Speech LLM via Self-Distilled Masked Diffusion Modelingcited by 1
  31. 2026StyleBench: Evaluating Speech Language Models on Conversational Speaking Style Controlcited by 1
  32. 2026TiCo: Time-Controllable Spoken Dialogue Modelcited by 1
  33. 2026VITA-QinYu: Expressive Spoken Language Model for Role-Playing and Singingcited by 1
  34. 2026PolySpeech-100: A Large-Scale Benchmark for Speech Understanding Across 100+ Languages and Dialectscited by 1
  35. 2026From Awareness to Adherence: Bridging the Context Gap in Spoken Dialogue Systems via Context-Aware Decodingcited by 1
  36. 2026Multi-turn Conversational AI from Text to Multimodal Interaction: Data, Models, Evaluation, and Open Challengescited by 1
  37. 2025GOAT-TTS: Expressive and Realistic Speech Generation via A Dual-Branch LLMcited by 0
  38. 2025RosettaSpeech: Zero-Shot Speech-to-Speech Translation without Parallel Speechcited by 0

50 most cited of 59 All 59 (JSON)