mlsanityData to 7 Oct 2026

Spirit LM: Interleaved Spoken and Written Language Model

arXiv 2402.05755 ↗First posted 8 Feb 2024cs.CL cs.SD eess.AScited by 123

On the map: Multimodal LLMs / vision-language models → MLLM training and visual instruction tuning

Abstract (arXiv)

We introduce Spirit LM, a foundation multimodal language model that freely mixes text and speech. Our model is based on a 7B pretrained text language model that we extend to the speech modality by continuously training it on text and speech units. Speech and text sequences are concatenated as a single stream of tokens, and trained with a word-level interleaving method using a small automatically-curated speech-text parallel corpus. Spirit LM comes in two versions: a Base version that uses speech phonetic units (HuBERT) and an Expressive version that models expressivity using pitch and style units in addition to the phonetic units. For both versions, the text is encoded with subword BPE tokens. The resulting model displays both the semantic abilities of text models and the expressive abilities of speech models. Additionally, we demonstrate that Spirit LM can learn new tasks in a few-shot fashion across modalities (i.e. ASR, TTS, Speech Classification). We make available model weights and inference code.

Key facts. Cited by 123 papers in the map (data through 7 Oct 2026). Placed in MLLM training and visual instruction tuning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 51 of its 69 references that are in the map. First posted on arXiv 8 Feb 2024.

Builds on

51 of 69 references.

Closest ancestors · 1

Same direction, newest first

  1. 2023SpeechGPT: Empowering Large Language Models with Intrinsic Cross-Modal Conversational Abilities18 May

Other references · 50

  1. 2017Attention Is All You Needcited by 59,932
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  4. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  5. 2020Measuring Massive Multitask Language Understandingcited by 7,768
  6. 2020Scaling Laws for Neural Language Modelscited by 6,411
  7. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
  8. 2017Neural Discrete Representation Learningcited by 4,650
  9. 2022Robust Speech Recognition via Large-Scale Weak Supervisioncited by 3,972
  10. 2020wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representationscited by 3,853
  11. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,634
  12. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,464
38 more
  1. 2019The Curious Case of Neural Text Degenerationcited by 2,881
  2. 2021HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Unitscited by 2,301
  3. 2023Code Llama: Open Foundation Models for Codecited by 2,105
  4. 2021WavLM: Large-Scale Self-Supervised Pre-Training for Full Stack Speech Processingcited by 1,609
  5. 2019Common Voice: A Massively-Multilingual Speech Corpuscited by 1,358
  6. 2020HiFi-GAN: Generative Adversarial Networks for Efficient and High Fidelity Speech Synthesiscited by 1,139
  7. 2021SoundStream: An End-to-End Neural Audio Codeccited by 756
  8. 2023Neural Codec Language Models are Zero-Shot Text to Speech Synthesizerscited by 737
  9. 2021SUPERB: Speech processing Universal PERformance Benchmarkcited by 652
  10. 2022AudioLM: a Language Modeling Approach to Audio Generationcited by 611
  11. 2020MLS: A Large-Scale Multilingual Dataset for Speech Researchcited by 467
  12. 2021VoxPopuli: A Large-Scale Multilingual Speech Corpus for Representation Learning, Semi-Supervised Learning and Interpretationcited by 455
  13. 2023Scaling Speech Technology to 1,000+ Languagescited by 401
  14. 2023Toxicity in ChatGPT: Analyzing Persona-assigned Language Modelscited by 356
  15. 2021W2v-BERT: Combining Contrastive Learning and Masked Language Modeling for Self-Supervised Speech Pre-Trainingcited by 340
  16. 2021Generative Spoken Language Modeling from Raw Audiocited by 336
  17. 2023AudioPaLM: A Large Language Model That Can Speak and Listencited by 333
  18. 2023Effective Long-Context Scaling of Foundation Modelscited by 301
  19. 2021Speech Resynthesis from Discrete Disentangled Self-Supervised Representationscited by 243
  20. 2023Seamless: Multilingual Expressive and Streaming Speech Translationcited by 190
  21. 2022"I'm sorry to hear that": Finding New Biases in Language Models with a Holistic Descriptor Datasetcited by 160
  22. 2021Direct speech-to-speech translation with discrete unitscited by 148
  23. 2021Textless Speech-to-Speech Translation on Real Datacited by 121
  24. 2021Text-Free Prosody-Aware Generative Spoken Language Modelingcited by 119
  25. 2020The Zero Resource Speech Benchmark 2021: Metrics and baselines for unsupervised spoken language modelingcited by 100
  26. 2023Textually Pretrained Speech Language Modelscited by 99
  27. 2023EXPRESSO: A Benchmark and Analysis of Discrete Expressive Speech Resynthesiscited by 93
  28. 2023Spoken Question Answering and Speech Continuation Using Spectrogram-Powered LLMcited by 92
  29. 2023Evaluating the Social Impact of Generative AI Systems in Systems and Societycited by 91
  30. 2023Voxtlm: unified decoder-only models for consolidating speech recognition/synthesis and speech/text continuation taskscited by 71
  31. 2018The Emotional Voices Database: Towards Controlling the Emotion Dimension in Voice Generation Systemscited by 61
  32. 2023SpeechPrompt v2: Prompt Tuning for Speech Classification Taskscited by 31
  33. 2021Textless Speech Emotion Conversion using Discrete and Decomposed Representationscited by 29
  34. 2023Toward Joint Language Modeling for Speech Units and Textcited by 27
  35. 2022Toxicity in Multilingual Machine Translation at Scalecited by 26
  36. 2023Generative Spoken Language Model based on continuous word-sized audio tokenscited by 19
  37. 2022Augmentation Invariant Discrete Representation for Generative Spoken Language Modelingcited by 17
  38. 2024MuTox: Universal MUltilingual Audio-based TOXicity Dataset and Zero-shot Detectorcited by 14

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 25; 2025: 65; 2026: 33.

123 papers.

Sort:

By area:

  1. 2024Moshi: a speech-text foundation model for real-time dialoguecited by 536
  2. 2025On The Landscape of Spoken Language Models: A Comprehensive Surveycited by 116
  3. 2024WavChat: A Survey of Spoken Dialogue Modelscited by 88
  4. 2024Recent Advances in Speech Language Models: A Surveycited by 85
  5. 2025LLaMA-Omni2: LLM-based Real-time Spoken Chatbot with Autoregressive Streaming Speech Synthesiscited by 75
  6. 2024Best-of-N Jailbreakingcited by 71
  7. 2024Beyond Turn-Based Interfaces: Synchronous LLMs as Full-Duplex Dialogue Agentscited by 69
  8. 2025Baichuan-Audio: A Unified Framework for End-to-End Speech Interactioncited by 68
  9. 2025Recent Advances in Discrete Speech Tokens: A Reviewcited by 42
  10. 2025Voxtralcited by 41
  11. 2024SALMONN-omni: A Codec-free LLM for Full-duplex Speech Understanding and Generationcited by 33
  12. 2025SALMONN-omni: A Standalone Speech LLM without Codec Injection for Full-duplex Conversationcited by 25
38 more
  1. 2025STITCH: Simultaneous Thinking and Talking with Chunked Reasoning for Spoken Language Modelscited by 25
  2. 2024Salmon: A Suite for Acoustic Language Model Evaluationcited by 24
  3. 2024SyllableLM: Learning Coarse Semantic Units for Speech Language Modelscited by 24
  4. 2024Align-SLM: Textless Spoken Language Models with Reinforcement Learning from AI Feedbackcited by 22
  5. 2025MTR-DuplexBench: Towards a Comprehensive Evaluation of Multi-Round Conversations for Full-Duplex Speech Language Modelscited by 20
  6. 2024Long-Form Speech Generation with Spoken Language Modelscited by 19
  7. 2025FocalCodec: Low-Bitrate Speech Coding via Focal Modulation Networkscited by 18
  8. 2025Stream-Omni: Simultaneous Multimodal Interactions with Large Language-Vision-Speech Modelcited by 17
  9. 2024SSR: Alignment-Aware Modality Connector for Speech Language Modelscited by 16
  10. 2025VoxEval: Benchmarking the Knowledge Understanding Capabilities of End-to-End Spoken Language Modelscited by 16
  11. 2025VocalNet: Speech LLM with Multi-Token Prediction for Faster and High-Quality Generationcited by 16
  12. 2025TASTE: Text-Aligned Speech Tokenization and Embedding for Spoken Language Modelingcited by 15
  13. 2025Watermarking Autoregressive Image Generationcited by 15
  14. 2025Slamming: Training a Speech Language Model on One GPU in a Daycited by 11
  15. 2025InSerter: Speech Instruction Following with Unsupervised Interleaved Pre-trainingcited by 11
  16. 2025JALMBench: Benchmarking Jailbreak Vulnerabilities in Audio Language Modelscited by 11
  17. 2024DiscreteSLU: A Large Language Model with Self-Supervised Discrete Speech Units for Spoken Language Understandingcited by 10
  18. 2024TouchTTS: An Embarrassingly Simple TTS Framework that Everyone Can Touchcited by 10
  19. 2025When Large Language Models Meet Speech: A Survey on Integration Approachescited by 9
  20. 2025OpusLM: A Family of Open Unified Speech Language Modelscited by 9
  21. 2026MOSS-Audio Technical Reportcited by 9
  22. 2024Improving Spoken Language Modeling with Phoneme Classification: A Simple Fine-tuning Approachcited by 8
  23. 2024DC-Spin: A Speaker-invariant Speech Tokenizer for Spoken Language Modelscited by 8
  24. 2024Contrastive Learning for Task-Independent SpeechLLM-Pretrainingcited by 8
  25. 2024Why Do Speech Language Models Fail to Generate Semantically Coherent Outputs? A Modality Evolving Perspectivecited by 8
  26. 2025Voila: Voice-Language Foundation Models for Real-Time Autonomous Interaction and Voice Role-Playcited by 8
  27. 2025SpeakStream: Streaming Text-to-Speech with Interleaved Datacited by 8
  28. 2025OpenS2S: Advancing Fully Open-Source End-to-End Empathetic Large Speech Language Modelcited by 8
  29. 2025Attacker's Noise Can Manipulate Your Audio-based LLM in the Real Worldcited by 8
  30. 2025FocalCodec-Stream: Streaming Low-Bitrate Speech Coding via Causal Distillationcited by 8
  31. 2024MERaLiON-AudioLLM: Bridging Audio and Language with Large Language Modelscited by 7
  32. 2025mSTEB: Massively Multilingual Evaluation of LLMs on Speech and Text Taskscited by 7
  33. 2025PredGen: Accelerated Inference of Large Language Models through Input-Time Speculation for Real-Time Speech Interactioncited by 7
  34. 2026Covo-Audio Technical Reportcited by 7
  35. 2025Overcoming Vocabulary Constraints with Pixel-level Fallbackcited by 6
  36. 2025Hearing to Translate: The Effectiveness of Speech Modality Integration into LLMscited by 6
  37. 2025SpidR: Learning Fast and Stable Linguistic Units for Spoken Language Models Without Supervisioncited by 6
  38. 2024Get Large Language Models Ready to Speak: A Late-fusion Approach for Speech Generationcited by 5

50 most cited of 123 All 123 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024Lyra: An Efficient and Speech-Centric Framework for Omni-Cognition
  2. 2026MoST: Mixing Speech and Text with Modality-Aware Mixture of Experts
  3. 2024MIO: A Foundation Model on Multimodal Tokens
  4. 2026Alignment Is All You Need: Instruction-Free Training for General Audio-Language Models
  5. 2024Model Composition for Multimodal Large Language Models
  6. 2025Soundwave: Less is More for Speech-Text Alignment in LLMs
  7. 2023Macaw-LLM: Multi-Modal Language Modeling with Image, Audio, Video, and Text Integration
  8. 2024Discrete Multimodal Transformers with a Pretrained Large Language Model for Mixed-Supervision Speech Processing