mlsanityData to 14 Oct 2026

Freeze-Omni: A Smart and Low Latency Speech-to-speech Dialogue Model with Frozen LLM

arXiv 2411.00774 ↗First posted 1 Nov 2024cs.SD cs.AI cs.CLcited by 141

On the map: Multimodal LLMs / vision-language models → MLLM training and visual instruction tuning

Abstract (arXiv)

Rapidly developing large language models (LLMs) have brought tremendous intelligent applications. Especially, the GPT-4o's excellent duplex speech interaction ability has brought impressive experience to users. Researchers have recently proposed several multi-modal LLMs in this direction that can achieve user-agent speech-to-speech conversations. This paper proposes a novel speech-text multimodal LLM architecture called Freeze-Omni. Our main contribution is that the speech input and output modalities can be easily connected to a textual LLM while keeping the LLM's parameters frozen throughout the training process. We design a three-stage training strategy for modeling both the speech input and output, enabling Freeze-Omni to obtain speech-to-speech conversation ability using text-speech paired data (such as ASR and TTS data) and only 60,000 multi-round text Q&A data on 8 GPUs. Moreover, we can effectively ensure that the intelligence of the Freeze-Omni in the speech modality is at the same level compared with that in the text modality of its backbone LLM, while achieving low latency end-to-end spoken response. In addition, we also designed a method to achieve duplex dialogue ability through multi-task training, giving Freeze-Omni a more natural style of dialogue ability between users and agents. In summary, Freeze-Omni holds great potential to conduct speech-to-speech dialogue based on a multimodal LLM under the condition of a frozen LLM, avoiding the catastrophic forgetting problem caused by limited data and training resources.

Key facts. Cited by 141 papers in the map (data through 14 Oct 2026). Placed in MLLM training and visual instruction tuning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 20 of its 27 references that are in the map. First posted on arXiv 1 Nov 2024.

Builds on

20 of 27 references.

Closest ancestors · 2

Same direction, newest first

  1. 2024VITA: Towards Open-Source Interactive Omni Multimodal LLM9 Aug
  2. 2023SpeechGPT: Empowering Large Language Models with Intrinsic Cross-Modal Conversational Abilities18 May

Other references · 18

  1. 2017Attention Is All You Needcited by 60,361
  2. 2023GPT-4 Technical Reportcited by 18,396
  3. 2024The Llama 3 Herd of Modelscited by 14,715
  4. 2020wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representationscited by 3,905
  5. 2017TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehensioncited by 2,955
  6. 2017Decoupled Weight Decay Regularizationcited by 1,831
  7. 2024Mixtral of Expertscited by 1,823
  8. 2024Moshi: a speech-text foundation model for real-time dialoguecited by 568
  9. 2024Qwen2-Audio Technical Reportcited by 560
  10. 2017AISHELL-1: An Open-Source Mandarin Speech Corpus and A Speech Recognition Baselinecited by 373
  11. 2024Mini-Omni: Language Models Can Hear, Talk While Thinking in Streamingcited by 210
  12. 2021WenetSpeech: A 10000+ Hours Multi-domain Mandarin Corpus for Speech Recognitioncited by 197
6 more
  1. 2024LLaMA-Omni: Seamless Speech Interaction with Large Language Modelscited by 173
  2. 2022Paraformer: Fast and Accurate Parallel Transformer for Non-autoregressive End-to-End Speech Recognitioncited by 144
  3. 2021WeNet: Production oriented Streaming and Non-streaming End-to-End Speech Recognition Toolkitcited by 125
  4. 2024VALL-E 2: Neural Codec Language Models are Human Parity Zero-Shot Text to Speech Synthesizerscited by 122
  5. 2023Spoken Question Answering and Speech Continuation Using Spectrogram-Powered LLMcited by 94
  6. 2023Fewer-token Neural Speech Codec with Time-invariant Codescited by 22

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 3; 2025: 71; 2026: 67.

141 papers.

Sort:

By area:

  1. 2025Kimi-Audio Technical Reportcited by 222
  2. 2025VITA-1.5: Towards GPT-4o Level Real-Time Vision and Speech Interactioncited by 197
  3. 2025On The Landscape of Spoken Language Models: A Comprehensive Surveycited by 118
  4. 2025Step-Audio 2 Technical Reportcited by 105
  5. 2025Full-Duplex-Bench: A Benchmark to Evaluate Full-duplex Spoken Dialogue Models on Turn-taking Capabilitiescited by 100
  6. 2024WavChat: A Survey of Spoken Dialogue Modelscited by 90
  7. 2025Step-Audio: Unified Understanding and Generation in Intelligent Speech Interactioncited by 83
  8. 2025MinMo: A Multimodal Large Language Model for Seamless Voice Interactioncited by 79
  9. 2025LLaMA-Omni2: LLM-based Real-time Spoken Chatbot with Autoregressive Streaming Speech Synthesiscited by 76
  10. 2025Baichuan-Audio: A Unified Framework for End-to-End Speech Interactioncited by 68
  11. 2026PersonaPlex: Voice and Role Control for Full Duplex Conversational Speech Modelscited by 63
  12. 2024InternLM-XComposer2.5-OmniLive: A Comprehensive Multimodal System for Long-term Streaming Video and Audio Interactionscited by 48
38 more
  1. 2025Recent Advances in Discrete Speech Tokens: A Reviewcited by 43
  2. 2025EmoVoice: LLM-based Emotional Text-To-Speech Model with Freestyle Text Promptingcited by 34
  3. 2025SALM-Duplex: Efficient and Direct Duplex Modeling for Speech-to-Speech Language Modelcited by 34
  4. 2025VITA-Audio: Fast Interleaved Cross-Modal Token Generation for Efficient Large Speech-Language Modelcited by 31
  5. 2025URO-Bench: Towards Comprehensive Evaluation for End-to-End Spoken Dialogue Modelscited by 30
  6. 2025InteractiveOmni: A Unified Omni-modal Model for Audio-Visual Multi-turn Dialoguecited by 29
  7. 2025SALMONN-omni: A Standalone Speech LLM without Codec Injection for Full-duplex Conversationcited by 27
  8. 2025AudioMarathon: A Comprehensive Benchmark for Long-Context Audio Understanding and Efficiency in Audio LLMscited by 22
  9. 2025MTR-DuplexBench: Towards a Comprehensive Evaluation of Multi-Round Conversations for Full-Duplex Speech Language Modelscited by 22
  10. 2025Mini-Omni-Reasoner: Token-Level Thinking-in-Speaking in Large Speech Modelscited by 21
  11. 2025NExT-OMNI: Towards Any-to-Any Omnimodal Foundation Models with Discrete Flow Matchingcited by 21
  12. 2025FD-Bench: A Full-Duplex Benchmarking Pipeline Designed for Full Duplex Spoken Dialogue Systemscited by 20
  13. 2025M2-omni: Advancing Omni-MLLM for Comprehensive Modality Support with Competitive Performancecited by 19
  14. 2025Fun-Audio-Chat Technical Reportcited by 19
  15. 2025Stream-Omni: Simultaneous Multimodal Interactions with Large Language-Vision-Speech Modelcited by 18
  16. 2025GOAL: Global-local Object Alignment Learningcited by 17
  17. 2025VocalBench: Benchmarking the Vocal Conversational Abilities for Speech Interaction Modelscited by 17
  18. 2025S2S-Arena: Evaluating Paralinguistic Instruction Following in Speech-to-Speech Modelscited by 16
  19. 2025VocalNet: Speech LLM with Multi-Token Prediction for Faster and High-Quality Generationcited by 16
  20. 2025JALMBench: Benchmarking Jailbreak Vulnerabilities in Audio Language Modelscited by 16
  21. 2025FlexDuo: A Pluggable System for Enabling Full-Duplex Capabilities in Speech Dialogue Systemscited by 15
  22. 2025WavReward: Spoken Dialogue Models With Generalist Reward Evaluatorscited by 15
  23. 2025FLEXI: Benchmarking Full-duplex Human-LLM Speech Interactioncited by 15
  24. 2026RIVER: A Real-Time Interaction Benchmark for Video LLMscited by 14
  25. 2025WildSpeech-Bench: Benchmarking End-to-End SpeechLLMs in the Wildcited by 13
  26. 2025TaDiCodec: Text-aware Diffusion Speech Tokenizer for Speech Language Modelingcited by 13
  27. 2025Easy Turn: Integrating Acoustic and Linguistic Modalities for Robust Turn-Taking in Full-Duplex Spoken Dialogue Systemscited by 13
  28. 2025Game-Time: Evaluating Temporal Dynamics in Spoken Language Modelscited by 13
  29. 2025Understanding the Modality Gap: An Empirical Study on the Speech-Text Alignment Mechanism of Large Speech Language Modelscited by 13
  30. 2026Omni-Diffusion: Unified Multimodal Understanding and Generation with Masked Discrete Diffusioncited by 13
  31. 2026MoshiRAG: Asynchronous Knowledge Retrieval for Full-Duplex Speech Language Modelscited by 12
  32. 2025FLM-Audio: Natural Monologues Improves Native Full-Duplex Chatbots via Dual Trainingcited by 11
  33. 2025OpenOmni: Advancing Open-Source Omnimodal Large Language Models with Progressive Multimodal Alignment and Real-Time Self-Aware Emotional Speech Synthesiscited by 10
  34. 2025From Turn-Taking to Synchronous Dialogue: A Survey of Full-Duplex Spoken Language Modelscited by 10
  35. 2026Multi-Faceted Interactivity Alignment in Full-Duplex Speech Modelscited by 10
  36. 2025When Continue Learning Meets Multimodal Large Language Model: A Surveycited by 9
  37. 2025OpenS2S: Advancing Fully Open-Source End-to-End Empathetic Large Speech Language Modelcited by 9
  38. 2025C3: A Bilingual Benchmark for Spoken Dialogue Models Exploring Challenges in Complex Conversationscited by 9

50 most cited of 141 All 141 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024Get Large Language Models Ready to Speak: A Late-fusion Approach for Speech Generation
  2. 2024SALMONN-omni: A Codec-free LLM for Full-duplex Speech Understanding and Generation
  3. 2024Continuous Speech Tokens Makes LLMs Robust Multi-Modality Learners
  4. 2025X-Talk: On the Underestimated Potential of Modular Speech-to-Speech Dialogue System
  5. 2024Mini-Omni2: Towards Open-source GPT-4o with Vision, Speech and Duplex Capabilities
  6. 2024Beyond Turn-Based Interfaces: Synchronous LLMs as Full-Duplex Dialogue Agents
  7. 2023LLaSM: Large Language and Speech Model
  8. 2025Spoken Conversational Agents with Large Language Models