mlsanityData to 7 Oct 2026

Mini-Omni: Language Models Can Hear, Talk While Thinking in Streaming

arXiv 2408.16725 ↗First posted 29 Aug 2024cs.AI cs.CL cs.HCcited by 204

On the map: LLM reasoning → Audio-language model reasoning

Abstract (arXiv)

Recent advances in language models have achieved significant progress. GPT-4o, as a new milestone, has enabled real-time conversations with humans, demonstrating near-human natural fluency. Such human-computer interaction necessitates models with the capability to perform reasoning directly with the audio modality and generate output in streaming. However, this remains beyond the reach of current academic models, as they typically depend on extra TTS systems for speech synthesis, resulting in undesirable latency. This paper introduces the Mini-Omni, an audio-based end-to-end conversational model, capable of real-time speech interaction. To achieve this capability, we propose a text-instructed speech generation method, along with batch-parallel strategies during inference to further boost the performance. Our method also helps to retain the original model's language capabilities with minimal degradation, enabling other works to establish real-time interaction capabilities. We call this training method "Any Model Can Talk". We also introduce the VoiceAssistant-400K dataset to fine-tune models optimized for speech output. To our best knowledge, Mini-Omni is the first fully end-to-end, open-source model for real-time speech interaction, offering valuable potential for future research.

Key facts. Cited by 204 papers in the map (data through 7 Oct 2026). Placed in Audio-language model reasoning, one of 17 directions in LLM reasoning. Builds on 22 of its 26 references that are in the map. First posted on arXiv 29 Aug 2024.

Builds on

22 of 26 references.

References · 22

  1. 2023Improved Baselines with Visual Instruction Tuningcited by 4,362
  2. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,337
  3. 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generationcited by 4,310
  4. 2022Robust Speech Recognition via Large-Scale Weak Supervisioncited by 3,972
  5. 2020wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representationscited by 3,853
  6. 2024Qwen2 Technical Reportcited by 2,076
  7. 2023Video-LLaMA: An Instruction-tuned Audio-Visual Language Model for Video Understandingcited by 1,461
  8. 2019LibriTTS: A Corpus Derived from LibriSpeech for Text-to-Speechcited by 639
  9. 2024Qwen2-Audio Technical Reportcited by 543
  10. 2023SpeechGPT: Empowering Large Language Models with Intrinsic Cross-Modal Conversational Abilitiescited by 518
  11. 2023SALMONN: Towards Generic Hearing Abilities for Large Language Modelscited by 492
  12. 2023Simple and Controllable Music Generationcited by 480
10 more
  1. 2020MLS: A Large-Scale Multilingual Dataset for Speech Researchcited by 467
  2. 2022BEATs: Audio Pre-Training with Acoustic Tokenizerscited by 378
  3. 2023Voicebox: Text-Guided Multilingual Universal Speech Generation at Scalecited by 373
  4. 2024VITA: Towards Open-Source Interactive Omni Multimodal LLMcited by 172
  5. 2023Speak, Read and Prompt: High-Fidelity Text-to-Speech with Minimal Supervisioncited by 167
  6. 2024WavLLM: Towards Robust and Adaptive Speech Large Language Modelcited by 107
  7. 2023InstructTTS: Modelling Expressive TTS in Discrete Latent Space with Natural Language Style Promptcited by 90
  8. 2023LauraGPT: Listen, Attend, Understand, and Regenerate Audio with GPTcited by 77
  9. 2024SpeechVerse: A Large-scale Generalizable Audio Language Modelcited by 62
  10. 2023Mega-TTS: Zero-Shot Text-to-Speech at Scale with Intrinsic Inductive Biascited by 51

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 23; 2025: 111; 2026: 70.

204 papers.

Sort:

By area:

  1. 2025Qwen2.5-Omni Technical Reportcited by 704
  2. 2025Kimi-Audio Technical Reportcited by 210
  3. 2024VoiceBench: Benchmarking LLM-Based Voice Assistantscited by 192
  4. 2025Multimodal Chain-of-Thought Reasoning: A Comprehensive Surveycited by 176
  5. 2024LLaMA-Omni: Seamless Speech Interaction with Large Language Modelscited by 169
  6. 2024Freeze-Omni: A Smart and Low Latency Speech-to-speech Dialogue Model with Frozen LLMcited by 135
  7. 2025On The Landscape of Spoken Language Models: A Comprehensive Surveycited by 116
  8. 2025Baichuan-Omni-1.5 Technical Reportcited by 107
  9. 2025VisuLogic: A Benchmark for Evaluating Visual Reasoning in Multi-modal Large Language Modelscited by 102
  10. 2025MMSU: A Massive Multi-task Spoken Language Understanding and Reasoning Benchmarkcited by 101
  11. 2025Step-Audio 2 Technical Reportcited by 101
  12. 2025Audio-Reasoner: Improving Reasoning Capability in Large Audio Language Modelscited by 94
38 more
  1. 2024WavChat: A Survey of Spoken Dialogue Modelscited by 88
  2. 2024Recent Advances in Speech Language Models: A Surveycited by 85
  3. 2025Step-Audio: Unified Understanding and Generation in Intelligent Speech Interactioncited by 83
  4. 2025LLaMA-Omni2: LLM-based Real-time Spoken Chatbot with Autoregressive Streaming Speech Synthesiscited by 75
  5. 2025HumanOmni: A Large Vision-Speech Language Model for Human-Centric Video Understandingcited by 58
  6. 2024OmniFlatten: An End-to-end GPT Model for Seamless Voice Conversationcited by 57
  7. 2026PersonaPlex: Voice and Role Control for Full Duplex Conversational Speech Modelscited by 57
  8. 2024InternLM-XComposer2.5-OmniLive: A Comprehensive Multimodal System for Long-term Streaming Video and Audio Interactionscited by 48
  9. 2025OmniMMI: A Comprehensive Multi-modal Interaction Benchmark in Streaming Video Contextscited by 48
  10. 2025Talking Turns: Benchmarking Audio Foundation Models on Turn-Taking Dynamicscited by 45
  11. 2024EMOVA: Empowering Language Models to See, Hear and Speak with Vivid Emotionscited by 44
  12. 2025Recent Advances in Discrete Speech Tokens: A Reviewcited by 42
  13. 2024SALMONN-omni: A Codec-free LLM for Full-duplex Speech Understanding and Generationcited by 33
  14. 2024AdvWave: Stealthy Adversarial Jailbreak Attack against Large Audio-Language Modelscited by 33
  15. 2025EmoVoice: LLM-based Emotional Text-To-Speech Model with Freestyle Text Promptingcited by 33
  16. 2025VITA-Audio: Fast Interleaved Cross-Modal Token Generation for Efficient Large Speech-Language Modelcited by 30
  17. 2025URO-Bench: Towards Comprehensive Evaluation for End-to-End Spoken Dialogue Modelscited by 29
  18. 2024Benchmarking Open-ended Audio Dialogue Understanding for Large Audio-Language Modelscited by 28
  19. 2024Next Token Prediction Towards Multimodal Intelligence: A Comprehensive Surveycited by 27
  20. 2025InteractiveOmni: A Unified Omni-modal Model for Audio-Visual Multi-turn Dialoguecited by 27
  21. 2024SocialMind: LLM-based Proactive AR Social Assistive System with Human-like Perception for In-situ Live Interactionscited by 25
  22. 2025SALMONN-omni: A Standalone Speech LLM without Codec Injection for Full-duplex Conversationcited by 25
  23. 2025STITCH: Simultaneous Thinking and Talking with Chunked Reasoning for Spoken Language Modelscited by 25
  24. 2024StreamChat: Chatting with Streaming Videocited by 23
  25. 2024Align-SLM: Textless Spoken Language Models with Reinforcement Learning from AI Feedbackcited by 22
  26. 2024Building a Taiwanese Mandarin Spoken Language Model: A First Attemptcited by 22
  27. 2025Uni-MoE-2.0-Omni: Scaling Language-Centric Omnimodal Large Model with Advanced MoE, Training and Datacited by 21
  28. 2024Lyra: An Efficient and Speech-Centric Framework for Omni-Cognitioncited by 20
  29. 2025VeOmni: Scaling Any Modality Model Training with Model-Centric Distributed Recipe Zoocited by 20
  30. 2025Mini-Omni-Reasoner: Token-Level Thinking-in-Speaking in Large Speech Modelscited by 20
  31. 2025MTR-DuplexBench: Towards a Comprehensive Evaluation of Multi-Round Conversations for Full-Duplex Speech Language Modelscited by 20
  32. 2025M2-omni: Advancing Omni-MLLM for Comprehensive Modality Support with Competitive Performancecited by 19
  33. 2025Audio-Aware Large Language Models as Judges for Speaking Stylescited by 17
  34. 2025Stream-Omni: Simultaneous Multimodal Interactions with Large Language-Vision-Speech Modelcited by 17
  35. 2025FD-Bench: A Full-Duplex Benchmarking Pipeline Designed for Full Duplex Spoken Dialogue Systemscited by 17
  36. 2024IntrinsicVoice: Empowering LLMs with Intrinsic Real-time Voice Interaction Abilitiescited by 16
  37. 2025S2S-Arena: Evaluating Paralinguistic Instruction Following in Speech-to-Speech Modelscited by 16
  38. 2025VocalNet: Speech LLM with Multi-Token Prediction for Faster and High-Quality Generationcited by 16

50 most cited of 204 All 204 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024Mini-Omni2: Towards Open-source GPT-4o with Vision, Speech and Duplex Capabilities
  2. 2026MiniCPM-o 4.5: Towards Real-Time Full-Duplex Omni-Modal Interaction
  3. 2025SHANKS: Simultaneous Hearing and Thinking for Spoken Language Models
  4. 2026MiniMind-O Technical Report: An Open Small-Scale Speech-Native Omni Model ↗
  5. 2025Thinking While Speaking: Inference-Time Knowledge Transfer for Responsive and Intelligent Conversational Voice Agents
  6. 2023Speak While You Think: Streaming Speech Synthesis During Text Generation
  7. 2023Teach me with a Whisper: Enhancing Large Language Models for Analyzing Spoken Transcripts using Speech Embeddings
  8. 2025MinMo: A Multimodal Large Language Model for Seamless Voice Interaction