mlsanityData to 14 Oct 2026

Textually Pretrained Speech Language Models

arXiv 2305.13009 ↗First posted 22 May 2023cs.CL cs.LG cs.SDcited by 101

Not on the map. Why?

Abstract (arXiv)

Speech language models (SpeechLMs) process and generate acoustic data only, without textual supervision. In this work, we propose TWIST, a method for training SpeechLMs using a warm-start from a pretrained textual language models. We show using both automatic and human evaluations that TWIST outperforms a cold-start SpeechLM across the board. We empirically analyze the effect of different model design choices such as the speech tokenizer, the pretrained textual model, and the dataset size. We find that model and dataset scale both play an important role in constructing better-performing SpeechLMs. Based on our observations, we present the largest (to the best of our knowledge) SpeechLM both in terms of number of parameters and training data. We additionally introduce two spoken versions of the StoryCloze textual benchmark to further improve model evaluation and advance future research in the field. We make speech samples, code and models publicly available: https://pages.cs.huji.ac.il/adiyoss-lab/twist/ .

Key facts. Cited by 101 papers in the map (data through 14 Oct 2026). Builds on 60 of its 79 references that are in the map. First posted on arXiv 22 May 2023.

Builds on

60 of 79 references.

References · 60

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  2. 2020Language Models are Few-Shot Learnerscited by 32,419
  3. 2022Training language models to follow instructions with human feedbackcited by 15,971
  4. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  5. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  6. 2018Representation Learning with Contrastive Predictive Codingcited by 7,777
  7. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,326
  8. 2021Zero-Shot Text-to-Image Generationcited by 4,288
  9. 2022Robust Speech Recognition via Large-Scale Weak Supervisioncited by 4,057
  10. 2020wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representationscited by 3,905
  11. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,512
  12. 2022Training Compute-Optimal Large Language Modelscited by 3,019
38 more
  1. 2015Neural Machine Translation of Rare Words with Subword Unitscited by 2,918
  2. 2021HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Unitscited by 2,342
  3. 2018Deep contextualized word representationscited by 2,290
  4. 2022BLOOM: A 176B-Parameter Open-Access Multilingual Language Modelcited by 1,984
  5. 2019Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelismcited by 1,868
  6. 2023Pythia: A Suite for Analyzing Large Language Models Across Training and Scalingcited by 1,786
  7. 2019Common Voice: A Massively-Multilingual Speech Corpuscited by 1,384
  8. 2022Scaling Autoregressive Models for Content-Rich Text-to-Image Generationcited by 1,295
  9. 2020HiFi-GAN: Generative Adversarial Networks for Efficient and High Fidelity Speech Synthesiscited by 1,154
  10. 2017Natural TTS Synthesis by Conditioning WaveNet on Mel Spectrogram Predictionscited by 804
  11. 2021SoundStream: An End-to-End Neural Audio Codeccited by 771
  12. 2020FastSpeech 2: Fast and High-Quality End-to-End Text to Speechcited by 717
  13. 2022High Fidelity Neural Audio Compressioncited by 687
  14. 2023Muse: Text-To-Image Generation via Masked Generative Transformerscited by 648
  15. 2022AudioLM: a Language Modeling Approach to Audio Generationcited by 616
  16. 2020MLS: A Large-Scale Multilingual Dataset for Speech Researchcited by 476
  17. 2019Libri-Light: A Benchmark for ASR with Limited or No Supervisioncited by 467
  18. 2021VoxPopuli: A Large-Scale Multilingual Speech Corpus for Representation Learning, Semi-Supervised Learning and Interpretationcited by 463
  19. 2023MusicLM: Generating Music From Textcited by 448
  20. 2022CodeRL: Mastering Code Generation through Pretrained Models and Deep Reinforcement Learningcited by 358
  21. 2021W2v-BERT: Combining Contrastive Learning and Masked Language Modeling for Self-Supervised Speech Pre-Trainingcited by 347
  22. 2021Generative Spoken Language Modeling from Raw Audiocited by 340
  23. 2018WaveGlow: A Flow-based Generative Network for Speech Synthesiscited by 309
  24. 2022AudioGen: Textually Guided Audio Generationcited by 295
  25. 2021Speech Resynthesis from Discrete Disentangled Self-Supervised Representationscited by 243
  26. 2020SLURP: A Spoken Language Understanding Resource Packagecited by 198
  27. 2021SpeechT5: Unified-Modal Encoder-Decoder Pre-Training for Spoken Language Processingcited by 180
  28. 2021Direct speech-to-speech translation with discrete unitscited by 149
  29. 2015Semantically Conditioned LSTM-based Natural Language Generation for Spoken Dialogue Systemscited by 134
  30. 2022Generative Spoken Dialogue Language Modelingcited by 126
  31. 2021Textless Speech-to-Speech Translation on Real Datacited by 121
  32. 2021Text-Free Prosody-Aware Generative Spoken Language Modelingcited by 119
  33. 2022mSLAM: Massively multilingual joint pre-training for speech and textcited by 108
  34. 2020The Zero Resource Speech Benchmark 2021: Metrics and baselines for unsupervised spoken language modelingcited by 101
  35. 2021The People's Speech: A Large-Scale Diverse English Speech Recognition Dataset for Commercial Usagecited by 101
  36. 2021SLAM: A Unified Encoder for Speech and Language Modeling via Speech-Text Joint Pre-Trainingcited by 69
  37. 2023SingSong: Generating musical accompaniments from singingcited by 55
  38. 2021A Survey on Spoken Language Understanding: Recent Advances and New Frontierscited by 53

50 most cited of 60 All 60 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 10; 2024: 31; 2025: 39; 2026: 21.

101 papers.

Sort:

By area:

  1. 2024Moshi: a speech-text foundation model for real-time dialoguecited by 568
  2. 2023AudioPaLM: A Large Language Model That Can Speak and Listencited by 343
  3. 2024Discrete Flow Matchingcited by 317
  4. 2024Spirit LM: Interleaved Spoken and Written Language Modelcited by 126
  5. 2025On The Landscape of Spoken Language Models: A Comprehensive Surveycited by 118
  6. 2024CALF: Aligning LLMs for Time Series Forecasting via Cross-modal Fine-Tuningcited by 108
  7. 2025Full-Duplex-Bench: A Benchmark to Evaluate Full-duplex Spoken Dialogue Models on Turn-taking Capabilitiescited by 100
  8. 2023Spoken Question Answering and Speech Continuation Using Spectrogram-Powered LLMcited by 94
  9. 2024Recent Advances in Speech Language Models: A Surveycited by 87
  10. 2023Voxtlm: unified decoder-only models for consolidating speech recognition/synthesis and speech/text continuation taskscited by 72
  11. 2023Diverse and Aligned Audio-to-Video Generation via Text-to-Video Model Adaptationcited by 69
  12. 2025Baichuan-Audio: A Unified Framework for End-to-End Speech Interactioncited by 68
38 more
  1. 2024The Larger the Better? Improved LLM Code-Generation via Budget Reallocationcited by 43
  2. 2025Recent Advances in Discrete Speech Tokens: A Reviewcited by 43
  3. 2024Let's Go Real Talk: Spoken Dialogue Model for Face-to-Face Conversationcited by 41
  4. 2024Autoregressive Diffusion Transformer for Text-to-Speech Synthesiscited by 34
  5. 2025InteractiveOmni: A Unified Omni-modal Model for Audio-Visual Multi-turn Dialoguecited by 29
  6. 2024Salmon: A Suite for Acoustic Language Model Evaluationcited by 24
  7. 2024SyllableLM: Learning Coarse Semantic Units for Speech Language Modelscited by 24
  8. 2023SpeechGen: Unlocking the Generative Power of Speech Language Models with Promptscited by 21
  9. 2024Sylber: Syllabic Embedding Representation of Speech from Raw Audiocited by 19
  10. 2024Long-Form Speech Generation with Spoken Language Modelscited by 19
  11. 2025FocalCodec: Low-Bitrate Speech Coding via Focal Modulation Networkscited by 18
  12. 2025Stream-Omni: Simultaneous Multimodal Interactions with Large Language-Vision-Speech Modelcited by 18
  13. 2024IntrinsicVoice: Empowering LLMs with Intrinsic Real-time Voice Interaction Abilitiescited by 17
  14. 2025Closing the Gap Between Text and Speech Understanding in LLMscited by 17
  15. 2024SSR: Alignment-Aware Modality Connector for Speech Language Modelscited by 16
  16. 2025TASTE: Text-Aligned Speech Tokenization and Embedding for Spoken Language Modelingcited by 16
  17. 2024Paralinguistics-Aware Speech-Empowered Large Language Models for Natural Conversationcited by 14
  18. 2025Scaling Analysis of Interleaved Speech-Text Language Modelscited by 14
  19. 2025WildSpeech-Bench: Benchmarking End-to-End SpeechLLMs in the Wildcited by 13
  20. 2024SpeechPrompt: Prompting Speech Language Models for Speech Processing Taskscited by 12
  21. 2024MSLM-S2ST: A Multitask Speech Language Model for Textless Speech-to-Speech Translation with Speaker Style Preservationcited by 11
  22. 2025Voila: Voice-Language Foundation Models for Real-Time Autonomous Interaction and Voice Role-Playcited by 10
  23. 2023GSQA: An End-to-End Model for Generative Spoken Question Answeringcited by 9
  24. 2024LAST: Language Model Aware Speech Tokenizationcited by 9
  25. 2024DC-Spin: A Speaker-invariant Speech Tokenizer for Spoken Language Modelscited by 9
  26. 2025OpenS2S: Advancing Fully Open-Source End-to-End Empathetic Large Speech Language Modelcited by 9
  27. 2025VStyle: A Benchmark for Voice Style Adaptation with Spoken Instructionscited by 9
  28. 2025FocalCodec-Stream: Streaming Low-Bitrate Speech Coding via Causal Distillationcited by 9
  29. 2024Why Do Speech Language Models Fail to Generate Semantically Coherent Outputs? A Modality Evolving Perspectivecited by 8
  30. 2025Discrete Tokenization for Multimodal LLMs: A Comprehensive Surveycited by 8
  31. 2025SpidR: Learning Fast and Stable Linguistic Units for Spoken Language Models Without Supervisioncited by 8
  32. 2025Flow-SLM: Joint Learning of Linguistic and Acoustic Information for Spoken Language Modelingcited by 7
  33. 2026Covo-Audio Technical Reportcited by 7
  34. 2023Towards General-Purpose Text-Instruction-Guided Voice Conversioncited by 6
  35. 2025PAST: Phonetic-Acoustic Speech Tokenizercited by 6
  36. 2023Exploring In-Context Learning of Textless Speech Language Model for Speech Classification Taskscited by 5
  37. 2024Transforming LLMs into Cross-modal and Cross-lingual Retrieval Systemscited by 5
  38. 2024DM-Codec: Distilling Multimodal Representations for Speech Tokenizationcited by 5

50 most cited of 101 All 101 (JSON)