mlsanityData to 14 Oct 2026

LLaMA-Omni: Seamless Speech Interaction with Large Language Models

arXiv 2409.06666 ↗First posted 10 Sep 2024cs.CL cs.AI cs.SDcited by 173

Not on the map. Why?

Abstract (arXiv)

Models like GPT-4o enable real-time interaction with large language models (LLMs) through speech, significantly enhancing user experience compared to traditional text-based interaction. However, there is still a lack of exploration on how to build speech interaction models based on open-source LLMs. To address this, we propose LLaMA-Omni, a novel model architecture designed for low-latency and high-quality speech interaction with LLMs. LLaMA-Omni integrates a pretrained speech encoder, a speech adaptor, an LLM, and a streaming speech decoder. It eliminates the need for speech transcription, and can simultaneously generate text and speech responses directly from speech instructions with extremely low latency. We build our model based on the latest Llama-3.1-8B-Instruct model. To align the model with speech interaction scenarios, we construct a dataset named InstructS2S-200K, which includes 200K speech instructions and corresponding speech responses. Experimental results show that compared to previous speech-language models, LLaMA-Omni provides better responses in both content and style, with a response latency as low as 226ms. Additionally, training LLaMA-Omni takes less than 3 days on just 4 GPUs, paving the way for the efficient development of speech-language models in the future.

Key facts. Cited by 173 papers in the map (data through 14 Oct 2026). Builds on 54 of its 66 references that are in the map. First posted on arXiv 10 Sep 2024.

Builds on

54 of 66 references.

References · 54

  1. 2017Attention Is All You Needcited by 60,361
  2. 2020Language Models are Few-Shot Learnerscited by 32,419
  3. 2024The Llama 3 Herd of Modelscited by 14,715
  4. 2014Neural Machine Translation by Jointly Learning to Align and Translatecited by 4,548
  5. 2022Robust Speech Recognition via Large-Scale Weak Supervisioncited by 4,057
  6. 2021HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Unitscited by 2,342
  7. 2020HiFi-GAN: Generative Adversarial Networks for Efficient and High Fidelity Speech Synthesiscited by 1,154
  8. 2023Enhancing Chat Language Models by Scaling High-quality Instructional Conversationscited by 839
  9. 2017Natural TTS Synthesis by Conditioning WaveNet on Mel Spectrogram Predictionscited by 804
  10. 2023Neural Codec Language Models are Zero-Shot Text to Speech Synthesizerscited by 752
  11. 2022AudioLM: a Language Modeling Approach to Audio Generationcited by 616
  12. 2023Qwen-Audio: Advancing Universal Audio Understanding via Unified Large-Scale Audio-Language Modelscited by 608
38 more
  1. 2021Conditional Variational Autoencoder with Adversarial Learning for End-to-End Text-to-Speechcited by 561
  2. 2024Qwen2-Audio Technical Reportcited by 560
  3. 2023SpeechGPT: Empowering Large Language Models with Intrinsic Cross-Modal Conversational Abilitiescited by 529
  4. 2023SALMONN: Towards Generic Hearing Abilities for Large Language Modelscited by 506
  5. 2022UTMOS: UTokyo-SaruLab System for VoiceMOS Challenge 2022cited by 390
  6. 2023AudioPaLM: A Large Language Model That Can Speak and Listencited by 343
  7. 2021Generative Spoken Language Modeling from Raw Audiocited by 340
  8. 2024CosyVoice: A Scalable Multilingual Zero-shot Text-to-speech Synthesizer based on Supervised Semantic Tokenscited by 334
  9. 2021Speech Resynthesis from Discrete Disentangled Self-Supervised Representationscited by 243
  10. 2023Pengi: An Audio Language Model for Audio Taskscited by 220
  11. 2024AnyGPT: Unified Multimodal LLM with Discrete Sequence Modelingcited by 217
  12. 2024Mini-Omni: Language Models Can Hear, Talk While Thinking in Streamingcited by 210
  13. 2023On decoder-only architecture for speech-to-text and large language model integrationcited by 145
  14. 2023Prompting Large Language Models with Speech Recognition Abilitiescited by 140
  15. 2022Generative Spoken Dialogue Language Modelingcited by 126
  16. 2024WavLLM: Towards Robust and Adaptive Speech Large Language Modelcited by 109
  17. 2019Monotonic Multihead Attentioncited by 93
  18. 2024An Embarrassingly Simple Approach for LLM with Strong ASR Capacitycited by 88
  19. 2023VioLA: Unified Codec Language Models for Speech Recognition, Synthesis, and Translationcited by 87
  20. 2017Online and Linear-Time Attention by Enforcing Monotonic Alignmentscited by 85
  21. 2023Connecting Speech Encoder and Large Language Model for ASRcited by 80
  22. 2023BLSP: Bootstrapping Language-Speech Pre-training via Behavior Alignment of Continuation Writingcited by 79
  23. 2024SpeechVerse: A Large-scale Generalizable Audio Language Modelcited by 63
  24. 2024Language Model Can Listen While Speakingcited by 63
  25. 2023AudioChatLlama: Towards General-Purpose Speech Abilities for LLMscited by 59
  26. 2023LLaSM: Large Language and Speech Modelcited by 58
  27. 2024SpeechGPT-Gen: Scaling Chain-of-Information Speech Generationcited by 37
  28. 2022Attention as a Guide for Simultaneous Speech Translationcited by 32
  29. 2023Hybrid Transducer and Attention based Encoder-Decoder Modeling for Speech-to-Text Taskscited by 20
  30. 2019Incremental Text-to-Speech Synthesis with Prefix-to-Prefix Frameworkcited by 15
  31. 2024Speech ReaLLM -- Real-time Streaming Speech Recognition with Multimodal LLMs by Teaching the Flow of Timecited by 15
  32. 2023Non-autoregressive Streaming Transformer for Simultaneous Translationcited by 12
  33. 2024LLaST: Improved End-to-end Speech Translation System Leveraged by Large Language Modelscited by 12
  34. 2023Integrating Pre-Trained Speech and Language Models for End-to-End Speech Recognitioncited by 10
  35. 2024A Non-autoregressive Generation Framework for End-to-End Simultaneous Speech-to-Speech Translationcited by 10
  36. 2020What the Future Brings: Investigating the Impact of Lookahead for Incremental Neural TTScited by 9
  37. 2023Speak While You Think: Streaming Speech Synthesis During Text Generationcited by 8
  38. 2020Incremental Text-to-Speech Synthesis Using Pseudo Lookahead with Large Pretrained Language Modelcited by 7

50 most cited of 54 All 54 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 17; 2025: 82; 2026: 74.

173 papers.

Sort:

By area:

  1. 2025Kimi-Audio Technical Reportcited by 222
  2. 2024VoiceBench: Benchmarking LLM-Based Voice Assistantscited by 200
  3. 2025VITA-1.5: Towards GPT-4o Level Real-Time Vision and Speech Interactioncited by 197
  4. 2024Freeze-Omni: A Smart and Low Latency Speech-to-speech Dialogue Model with Frozen LLMcited by 141
  5. 2025On The Landscape of Spoken Language Models: A Comprehensive Surveycited by 118
  6. 2025Baichuan-Omni-1.5 Technical Reportcited by 111
  7. 2025VisuLogic: A Benchmark for Evaluating Visual Reasoning in Multi-modal Large Language Modelscited by 107
  8. 2025Step-Audio 2 Technical Reportcited by 105
  9. 2024WavChat: A Survey of Spoken Dialogue Modelscited by 90
  10. 2024Recent Advances in Speech Language Models: A Surveycited by 87
  11. 2025Ola: Pushing the Frontiers of Omni-Modal Language Modelcited by 83
  12. 2025Step-Audio: Unified Understanding and Generation in Intelligent Speech Interactioncited by 83
38 more
  1. 2025Baichuan-Audio: A Unified Framework for End-to-End Speech Interactioncited by 68
  2. 2025HumanOmni: A Large Vision-Speech Language Model for Human-Centric Video Understandingcited by 60
  3. 2024EMOVA: Empowering Language Models to See, Hear and Speak with Vivid Emotionscited by 45
  4. 2025Recent Advances in Discrete Speech Tokens: A Reviewcited by 43
  5. 2025Voxtralcited by 43
  6. 2025SAKURA: On the Multi-hop Reasoning of Large Audio-Language Models Based on Speech and Audio Informationcited by 42
  7. 2024AdvWave: Stealthy Adversarial Jailbreak Attack against Large Audio-Language Modelscited by 38
  8. 2024SALMONN-omni: A Codec-free LLM for Full-duplex Speech Understanding and Generationcited by 33
  9. 2025VITA-Audio: Fast Interleaved Cross-Modal Token Generation for Efficient Large Speech-Language Modelcited by 31
  10. 2025InteractiveOmni: A Unified Omni-modal Model for Audio-Visual Multi-turn Dialoguecited by 29
  11. 2024Next Token Prediction Towards Multimodal Intelligence: A Comprehensive Surveycited by 27
  12. 2025SALMONN-omni: A Standalone Speech LLM without Codec Injection for Full-duplex Conversationcited by 27
  13. 2025STITCH: Simultaneous Thinking and Talking with Chunked Reasoning for Spoken Language Modelscited by 26
  14. 2025A Survey of Token Compression for Efficient Multimodal Large Language Modelscited by 25
  15. 2024MIO: A Foundation Model on Multimodal Tokenscited by 24
  16. 2025Audio Jailbreak: An Open Comprehensive Benchmark for Jailbreaking Large Audio-Language Modelscited by 21
  17. 2025Mini-Omni-Reasoner: Token-Level Thinking-in-Speaking in Large Speech Modelscited by 21
  18. 2025NExT-OMNI: Towards Any-to-Any Omnimodal Foundation Models with Discrete Flow Matchingcited by 21
  19. 2025AudioJailbreak: Jailbreak Attacks against End-to-End Large Audio-Language Modelscited by 18
  20. 2025Stream-Omni: Simultaneous Multimodal Interactions with Large Language-Vision-Speech Modelcited by 18
  21. 2024IntrinsicVoice: Empowering LLMs with Intrinsic Real-time Voice Interaction Abilitiescited by 17
  22. 2025VocalBench: Benchmarking the Vocal Conversational Abilities for Speech Interaction Modelscited by 17
  23. 2025VocalNet: Speech LLM with Multi-Token Prediction for Faster and High-Quality Generationcited by 16
  24. 2025TASTE: Text-Aligned Speech Tokenization and Embedding for Spoken Language Modelingcited by 16
  25. 2025JALMBench: Benchmarking Jailbreak Vulnerabilities in Audio Language Modelscited by 16
  26. 2025WavReward: Spoken Dialogue Models With Generalist Reward Evaluatorscited by 15
  27. 2025Scaling Analysis of Interleaved Speech-Text Language Modelscited by 14
  28. 2025SpeechR: A Benchmark for Speech Reasoning in Large Audio-Language Modelscited by 14
  29. 2025WildSpeech-Bench: Benchmarking End-to-End SpeechLLMs in the Wildcited by 13
  30. 2025TaDiCodec: Text-aware Diffusion Speech Tokenizer for Speech Language Modelingcited by 13
  31. 2024Typhoon 2: A Family of Open Text and Multimodal Thai Large Language Modelscited by 12
  32. 2025Nexus: An Omni-Perceptive And -Interactive Model for Language, Audio, And Visioncited by 12
  33. 2026MoshiRAG: Asynchronous Knowledge Retrieval for Full-Duplex Speech Language Modelscited by 12
  34. 2025Adaptive Audio-Visual Speech Recognition via Matryoshka-Based Multimodal LLMscited by 11
  35. 2025Omni-Reward: Towards Generalist Omni-Modal Reward Modeling with Free-Form Preferencescited by 11
  36. 2026CORD: Bridging the Audio-Text Reasoning Gap via Weighted On-policy Cross-modal Distillationcited by 11
  37. 2025OpenOmni: Advancing Open-Source Omnimodal Large Language Models with Progressive Multimodal Alignment and Real-Time Self-Aware Emotional Speech Synthesiscited by 10
  38. 2025Scaling and Enhancing LLM-based AVSR: A Sparse Mixture of Projectors Approachcited by 10

50 most cited of 173 All 173 (JSON)