mlsanityData to 14 Oct 2026

LLaMA-Omni2: LLM-based Real-time Spoken Chatbot with Autoregressive Streaming Speech Synthesis

arXiv 2505.02625 ↗First posted 5 May 2025cs.CL cs.AI cs.SDcited by 76

Not on the map. Why?

Abstract (arXiv)

Real-time, intelligent, and natural speech interaction is an essential part of the next-generation human-computer interaction. Recent advancements have showcased the potential of building intelligent spoken chatbots based on large language models (LLMs). In this paper, we introduce LLaMA-Omni 2, a series of speech language models (SpeechLMs) ranging from 0.5B to 14B parameters, capable of achieving high-quality real-time speech interaction. LLaMA-Omni 2 is built upon the Qwen2.5 series models, integrating a speech encoder and an autoregressive streaming speech decoder. Despite being trained on only 200K multi-turn speech dialogue samples, LLaMA-Omni 2 demonstrates strong performance on several spoken question answering and speech instruction following benchmarks, surpassing previous state-of-the-art SpeechLMs like GLM-4-Voice, which was trained on millions of hours of speech data.

Key facts. Cited by 76 papers in the map (data through 14 Oct 2026). Builds on 32 of its 51 references that are in the map. First posted on arXiv 5 May 2025.

Builds on

32 of 51 references.

References · 32

  1. 2017Attention Is All You Needcited by 60,361
  2. 2024The Llama 3 Herd of Modelscited by 14,715
  3. 2022Robust Speech Recognition via Large-Scale Weak Supervisioncited by 4,057
  4. 2020HiFi-GAN: Generative Adversarial Networks for Efficient and High Fidelity Speech Synthesiscited by 1,154
  5. 2023Enhancing Chat Language Models by Scaling High-quality Instructional Conversationscited by 839
  6. 2023Qwen-Audio: Advancing Universal Audio Understanding via Unified Large-Scale Audio-Language Modelscited by 608
  7. 2024Moshi: a speech-text foundation model for real-time dialoguecited by 568
  8. 2023SpeechGPT: Empowering Large Language Models with Intrinsic Cross-Modal Conversational Abilitiescited by 529
  9. 2023Finite Scalar Quantization: VQ-VAE Made Simplecited by 440
  10. 2022UTMOS: UTokyo-SaruLab System for VoiceMOS Challenge 2022cited by 390
  11. 2023AudioPaLM: A Large Language Model That Can Speak and Listencited by 343
  12. 2024GLM-4-Voice: Towards Intelligent and Human-Like End-to-End Spoken Chatbotcited by 224
20 more
  1. 2024Mini-Omni: Language Models Can Hear, Talk While Thinking in Streamingcited by 210
  2. 2023On decoder-only architecture for speech-to-text and large language model integrationcited by 145
  3. 2024Freeze-Omni: A Smart and Low Latency Speech-to-speech Dialogue Model with Frozen LLMcited by 141
  4. 2024Spirit LM: Interleaved Spoken and Written Language Modelcited by 126
  5. 2024FunAudioLLM: Voice Understanding and Generation Foundation Models for Natural Interaction Between Humans and LLMscited by 122
  6. 2023Spoken Question Answering and Speech Continuation Using Spectrogram-Powered LLMcited by 94
  7. 2024WavChat: A Survey of Spoken Dialogue Modelscited by 90
  8. 2023Connecting Speech Encoder and Large Language Model for ASRcited by 80
  9. 2023BLSP: Bootstrapping Language-Speech Pre-training via Behavior Alignment of Continuation Writingcited by 79
  10. 2025MinMo: A Multimodal Large Language Model for Seamless Voice Interactioncited by 79
  11. 2024Language Model Can Listen While Speakingcited by 63
  12. 2024OmniFlatten: An End-to-end GPT Model for Seamless Voice Conversationcited by 60
  13. 2023AudioChatLlama: Towards General-Purpose Speech Abilities for LLMscited by 59
  14. 2024SLAM-Omni: Timbre-Controllable Voice Interaction System with Single-Stage Trainingcited by 59
  15. 2024Scaling Speech-Text Pre-training with Synthetic Interleaved Datacited by 47
  16. 2024SpeechGPT-Gen: Scaling Chain-of-Information Speech Generationcited by 37
  17. 2018The State of Speech in HCI: Trends, Themes and Challengescited by 21
  18. 2024IntrinsicVoice: Empowering LLMs with Intrinsic Real-time Voice Interaction Abilitiescited by 17
  19. 2024A Non-autoregressive Generation Framework for End-to-End Simultaneous Speech-to-Speech Translationcited by 10
  20. 2024Overcoming Non-monotonicity in Transducer-based Streaming Generationcited by 4

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 25; 2026: 51.

76 papers.

Sort:

By area:

  1. 2026PersonaPlex: Voice and Role Control for Full Duplex Conversational Speech Modelscited by 63
  2. 2025InteractiveOmni: A Unified Omni-modal Model for Audio-Visual Multi-turn Dialoguecited by 29
  3. 2025STITCH: Simultaneous Thinking and Talking with Chunked Reasoning for Spoken Language Modelscited by 26
  4. 2025Stream-Omni: Simultaneous Multimodal Interactions with Large Language-Vision-Speech Modelcited by 18
  5. 2025VocalBench: Benchmarking the Vocal Conversational Abilities for Speech Interaction Modelscited by 17
  6. 2025WavReward: Spoken Dialogue Models With Generalist Reward Evaluatorscited by 15
  7. 2026MoshiRAG: Asynchronous Knowledge Retrieval for Full-Duplex Speech Language Modelscited by 12
  8. 2025EchoMind: An Interrelated Multi-level Benchmark for Evaluating Empathetic Speech Language Modelscited by 11
  9. 2026Maximizing Local Entropy Where It Matters: Prefix-Aware Localized LLM Unlearningcited by 10
  10. 2025OpenS2S: Advancing Fully Open-Source End-to-End Empathetic Large Speech Language Modelcited by 9
  11. 2025Mind-Paced Speaking: A Dual-Brain Approach to Real-Time Reasoning in Spoken Language Modelscited by 9
  12. 2025VocalBench-DF: A Benchmark for Evaluating Speech LLM Robustness to Disfluencycited by 8
38 more
  1. 2026HalluAudio: A Comprehensive Benchmark for Hallucination Detection in Large Audio-Language Modelscited by 8
  2. 2026Covo-Audio Technical Reportcited by 7
  3. 2025DrVoice: Parallel Speech-Text Voice Conversation Model via Dual-Resolution Speech Representationscited by 6
  4. 2026LTS-VoiceAgent: A Listen-Think-Speak Framework for Efficient Streaming Voice Interaction via Semantic Triggering and Incremental Reasoningcited by 6
  5. 2026WavBench: Benchmarking Reasoning, Colloquialism, and Paralinguistics for End-to-End Spoken Dialogue Modelscited by 6
  6. 2026BayLing-Duplex: Native Full-Duplex Speech Dialogue with a Single Autoregressive LLMcited by 6
  7. 2026Moment-Video: Diagnosing Temporal Fidelity of Video MLLMs on Momentary Visual Eventscited by 5
  8. 2025ChipChat: Low-Latency Cascaded Conversational Agent in MLXcited by 4
  9. 2025EchoX: Towards Mitigating Acoustic-Semantic Gap via Echo Training for Speech-to-Speech LLMscited by 4
  10. 2025StableToken: A Noise-Robust Semantic Speech Tokenizer for Resilient SpeechLLMscited by 4
  11. 2026Audio Interaction Modelcited by 4
  12. 2026Watch, Remember, Reason: Human-View Video Understanding with MLLMscited by 4
  13. 2026From Static Inference to Dynamic Interaction: A Survey of Streaming Large Language Modelscited by 3
  14. 2026Hijacking Large Audio-Language Models via Context-Agnostic and Imperceptible Auditory Prompt Injectioncited by 3
  15. 2025FastLongSpeech: Enhancing Large Speech-Language Models for Efficient Long-Speech Processingcited by 2
  16. 2025TELEVAL: A Benchmark Designed for Spoken Language Models in Chinese Interactive Scenarioscited by 2
  17. 2025GOAT-SLM: A Spoken Language Model with Paralinguistic and Speaker Characteristic Awarenesscited by 2
  18. 2025Training-Free Multimodal Large Language Model Orchestrationcited by 2
  19. 2025UltraVoice: Scaling Fine-Grained Style-Controlled Speech Conversations for Spoken Dialogue Modelscited by 2
  20. 2025Protecting Bystander Privacy via Selective Hearing in Audio LLMscited by 2
  21. 2026MoST: Mixing Speech and Text with Modality-Aware Mixture of Expertscited by 2
  22. 2026AEQ-Bench: Measuring Empathy of Omni-Modal Large Modelscited by 2
  23. 2026SonicBench: Dissecting the Physical Perception Bottleneck in Large Audio Language Modelscited by 2
  24. 2026Ex-Omni: Enabling 3D Facial Animation Generation for Omni-modal Large Language Modelscited by 2
  25. 2026GRM: Utility-Aware Jailbreak Attacks on Audio LLMs via Gradient-Ratio Maskingcited by 2
  26. 2026SpeechParaling-Bench: A Comprehensive Benchmark for Paralinguistic-Aware Speech Generationcited by 2
  27. 2026Minimizing Modality Gap from the Input Side: Your Speech LLM Can Be a Prosody-Aware Text LLMcited by 2
  28. 2026Real-Time Voice AI Hears but Does Not Listencited by 2
  29. 2026Vox-Infinity: Benchmarking the Limits of Long-Context Spoken Language Modelscited by 2
  30. 2025Efficient Speech Language Modeling via Energy Distance in Continuous Latent Spacecited by 1
  31. 2025Towards Efficient Speech-Text Jointly Decoding within One Speech Language Modelcited by 1
  32. 2025JoyTTS: LLM-based Spoken Chatbot With Voice Cloningcited by 1
  33. 2025Enhancing Speech Large Language Models through Reinforced Behavior Alignmentcited by 1
  34. 2025Acoustic-based Gender Differentiation in Speech-aware Language Modelscited by 1
  35. 2026Pardon? Evaluating Conversational Repair in Large Audio-Language Modelscited by 1
  36. 2026A2-LLM: An End-to-end Conversational Audio Avatar Large Language Modelcited by 1
  37. 2026VocalNet-MDM: Accelerating Streaming Speech LLM via Self-Distilled Masked Diffusion Modelingcited by 1
  38. 2026Language-Aware Distillation for Multilingual Instruction-Following Speech LLMs with ASR-Only Supervisioncited by 1

50 most cited of 76 All 76 (JSON)