mlsanityData to 7 Oct 2026

Moshi: a speech-text foundation model for real-time dialogue

arXiv 2410.00037 ↗First posted 17 Sep 2024eess.AS cs.AI cs.CLcited by 536

Not on the map. Why?

Cited by 48 in LLM reasoning, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

We introduce Moshi, a speech-text foundation model and full-duplex spoken dialogue framework. Current systems for spoken dialogue rely on pipelines of independent components, namely voice activity detection, speech recognition, textual dialogue and text-to-speech. Such frameworks cannot emulate the experience of real conversations. First, their complexity induces a latency of several seconds between interactions. Second, text being the intermediate modality for dialogue, non-linguistic information that modifies meaning -- such as emotion or non-speech sounds -- is lost in the interaction. Finally, they rely on a segmentation into speaker turns, which does not take into account overlapping speech, interruptions and interjections. Moshi solves these independent issues altogether by casting spoken dialogue as speech-to-speech generation. Starting from a text language model backbone, Moshi generates speech as tokens from the residual quantizer of a neural audio codec, while modeling separately its own speech and that of the user into parallel streams. This allows for the removal of explicit speaker turns, and the modeling of arbitrary conversational dynamics. We moreover extend the hierarchical semantic-to-acoustic token generation of previous work to first predict time-aligned text tokens as a prefix to audio tokens. Not only this "Inner Monologue" method significantly improves the linguistic quality of generated speech, but we also illustrate how it can provide streaming speech recognition and text-to-speech. Our resulting model is the first real-time full-duplex spoken large language model, with a theoretical latency of 160ms, 200ms in practice, and is available at https://github.com/kyutai-labs/moshi.

Key facts. Cited by 536 papers in the map (data through 7 Oct 2026). Builds on 67 of its 117 references that are in the map. First posted on arXiv 17 Sep 2024.

Builds on

67 of 117 references.

References · 67

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2020Language Models are Few-Shot Learnerscited by 32,199
  4. 2020Denoising Diffusion Probabilistic Modelscited by 20,851
  5. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  6. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  7. 2020Measuring Massive Multitask Language Understandingcited by 7,768
  8. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
  9. 2016SGDR: Stochastic Gradient Descent with Warm Restartscited by 4,993
  10. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,710
  11. 2017Neural Discrete Representation Learningcited by 4,650
  12. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,018
38 more
  1. 2022Robust Speech Recognition via Large-Scale Weak Supervisioncited by 3,972
  2. 2020wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representationscited by 3,853
  3. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,611
  4. 2016Gaussian Error Linear Units (GELUs)cited by 3,431
  5. 2024Gemini 1.5: Unlocking multimodal understanding across millions of tokens of contextcited by 3,175
  6. 2022Training Compute-Optimal Large Language Modelscited by 2,927
  7. 2017TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehensioncited by 2,895
  8. 2019The Curious Case of Neural Text Degenerationcited by 2,881
  9. 2018Can a Suit of Armor Conduct Electricity? A New Dataset for Open Book Question Answeringcited by 2,114
  10. 2018SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processingcited by 1,973
  11. 2020GLU Variants Improve Transformercited by 1,724
  12. 2017Decoupled Weight Decay Regularizationcited by 1,722
  13. 2016WaveNet: A Generative Model for Raw Audiocited by 1,710
  14. 2019Root Mean Square Layer Normalizationcited by 1,425
  15. 2015Fast and Accurate Deep Network Learning by Exponential Linear Units (ELUs)cited by 1,061
  16. 2016Bag of Tricks for Efficient Text Classificationcited by 869
  17. 2019DialoGPT: Large-Scale Generative Pre-training for Conversational Response Generationcited by 842
  18. 2023Gemini: A Family of Highly Capable Multimodal Modelscited by 810
  19. 2022Quantifying Memorization Across Neural Language Modelscited by 755
  20. 2023Neural Codec Language Models are Zero-Shot Text to Speech Synthesizerscited by 737
  21. 2022High Fidelity Neural Audio Compressioncited by 673
  22. 2023A Watermark for Large Language Modelscited by 667
  23. 2023AudioLDM: Text-to-Audio Generation with Latent Diffusion Modelscited by 592
  24. 2016Weight Normalization: A Simple Reparameterization to Accelerate Training of Deep Neural Networkscited by 541
  25. 2023The Falcon Series of Open Language Modelscited by 526
  26. 2023Simple and Controllable Music Generationcited by 480
  27. 2021A White Paper on Neural Network Quantizationcited by 453
  28. 2023High-Fidelity Audio Compression with Improved RVQGANcited by 423
  29. 2018HiDDeN: Hiding Data With Deep Networkscited by 384
  30. 2024QuIP#: Even Better LLM Quantization with Hadamard Incoherence and Lattice Codebookscited by 354
  31. 2021Generative Spoken Language Modeling from Raw Audiocited by 336
  32. 2023AudioPaLM: A Large Language Model That Can Speak and Listencited by 333
  33. 2022The case for 4-bit precision: k-bit Inference Scaling Lawscited by 247
  34. 2023Google USM: Scaling Automatic Speech Recognition Beyond 100 Languagescited by 228
  35. 2024NaturalSpeech 3: Zero-Shot Speech Synthesis with Factorized Codec and Diffusion Modelscited by 222
  36. 2024Stable Audio Opencited by 192
  37. 2023MEGABYTE: Predicting Million-byte Sequences with Multiscale Transformerscited by 164
  38. 2023UniAudio: An Audio Foundation Model Toward Universal Audio Generationcited by 127

50 most cited of 67 All 67 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 30; 2025: 228; 2026: 278.

536 papers.

Sort:

By area:

  1. 2025SmolVLA: A Vision-Language-Action Model for Affordable and Efficient Roboticscited by 507
  2. 2025Kimi-Audio Technical Reportcited by 210
  3. 2024VoiceBench: Benchmarking LLM-Based Voice Assistantscited by 192
  4. 2025VITA-1.5: Towards GPT-4o Level Real-Time Vision and Speech Interactioncited by 191
  5. 2024Freeze-Omni: A Smart and Low Latency Speech-to-speech Dialogue Model with Frozen LLMcited by 135
  6. 2025LLM Post-Training: A Deep Dive into Reasoning Large Language Modelscited by 129
  7. 2025On The Landscape of Spoken Language Models: A Comprehensive Surveycited by 116
  8. 2025Spark-TTS: An Efficient LLM-Based Text-to-Speech Model with Single-Stream Decoupled Speech Tokenscited by 109
  9. 2025Baichuan-Omni-1.5 Technical Reportcited by 107
  10. 2026Qwen3-TTS Technical Reportcited by 106
  11. 2025VisuLogic: A Benchmark for Evaluating Visual Reasoning in Multi-modal Large Language Modelscited by 102
  12. 2025MMSU: A Massive Multi-task Spoken Language Understanding and Reasoning Benchmarkcited by 101
38 more
  1. 2025Step-Audio 2 Technical Reportcited by 101
  2. 2025Audio-Reasoner: Improving Reasoning Capability in Large Audio Language Modelscited by 94
  3. 2025Full-Duplex-Bench: A Benchmark to Evaluate Full-duplex Spoken Dialogue Models on Turn-taking Capabilitiescited by 94
  4. 2024WavChat: A Survey of Spoken Dialogue Modelscited by 88
  5. 2024Recent Advances in Speech Language Models: A Surveycited by 85
  6. 2025Step-Audio: Unified Understanding and Generation in Intelligent Speech Interactioncited by 83
  7. 2025LLaMA-Omni2: LLM-based Real-time Spoken Chatbot with Autoregressive Streaming Speech Synthesiscited by 75
  8. 2024Multimodal Alignment and Fusion: A Surveycited by 73
  9. 2025Llasa: Scaling Train-Time and Inference-Time Compute for Llama-based Speech Synthesiscited by 69
  10. 2025Baichuan-Audio: A Unified Framework for End-to-End Speech Interactioncited by 68
  11. 2025MiMo-Audio: Audio Language Models are Few-Shot Learnerscited by 58
  12. 2024OmniFlatten: An End-to-end GPT Model for Seamless Voice Conversationcited by 57
  13. 2025Embodied AI Agents: Modeling the Worldcited by 57
  14. 2026PersonaPlex: Voice and Role Control for Full Duplex Conversational Speech Modelscited by 57
  15. 2024Scaling Transformers for Low-Bitrate High-Quality Speech Codingcited by 48
  16. 2025OmniMMI: A Comprehensive Multi-modal Interaction Benchmark in Streaming Video Contextscited by 48
  17. 2024Multimodal Latent Language Modeling with Next-Token Diffusioncited by 47
  18. 2025Scaling Laws for Native Multimodal Modelscited by 47
  19. 2025DeSTA2.5-Audio: Toward General-Purpose Large Audio Language Model with Self-Generated Cross-Modal Alignmentcited by 47
  20. 2025Talking Turns: Benchmarking Audio Foundation Models on Turn-Taking Dynamicscited by 45
  21. 2024EMOVA: Empowering Language Models to See, Hear and Speak with Vivid Emotionscited by 44
  22. 2025Recent Advances in Discrete Speech Tokens: A Reviewcited by 42
  23. 2025SAKURA: On the Multi-hop Reasoning of Large Audio-Language Models Based on Speech and Audio Informationcited by 42
  24. 2025ViSpeak: Visual Instruction Feedback in Streaming Videoscited by 38
  25. 2025Towards Holistic Evaluation of Large Audio-Language Models: A Comprehensive Surveycited by 38
  26. 2024SALMONN-omni: A Codec-free LLM for Full-duplex Speech Understanding and Generationcited by 33
  27. 2025SALM-Duplex: Efficient and Direct Duplex Modeling for Speech-to-Speech Language Modelcited by 32
  28. 2025VITA-Audio: Fast Interleaved Cross-Modal Token Generation for Efficient Large Speech-Language Modelcited by 30
  29. 2026LongCat-Next: Lexicalizing Modalities as Discrete Tokenscited by 30
  30. 2026Full-Duplex-Bench-v3: Benchmarking Tool Use for Full-Duplex Voice Agents Under Real-World Disfluencycited by 29
  31. 2024Benchmarking Open-ended Audio Dialogue Understanding for Large Audio-Language Modelscited by 28
  32. 2025ARTalk: Speech-Driven 3D Head Animation via Autoregressive Modelcited by 28
  33. 2024Next Token Prediction Towards Multimodal Intelligence: A Comprehensive Surveycited by 27
  34. 2025InteractiveOmni: A Unified Omni-modal Model for Audio-Visual Multi-turn Dialoguecited by 27
  35. 2025SALMONN-omni: A Standalone Speech LLM without Codec Injection for Full-duplex Conversationcited by 25
  36. 2025STITCH: Simultaneous Thinking and Talking with Chunked Reasoning for Spoken Language Modelscited by 25
  37. 2024SyllableLM: Learning Coarse Semantic Units for Speech Language Modelscited by 24
  38. 2025Streaming Sequence-to-Sequence Learning with Delayed Streams Modelingcited by 24

50 most cited of 536 All 536 (JSON)