mlsanityData to 14 Oct 2026

Toward Joint Language Modeling for Speech Units and Text

arXiv 2310.08715 ↗First posted 12 Oct 2023cs.CL cs.AI cs.SDcited by 28

Not on the map. Why?

Abstract (arXiv)

Speech and text are two major forms of human language. The research community has been focusing on mapping speech to text or vice versa for many years. However, in the field of language modeling, very little effort has been made to model them jointly. In light of this, we explore joint language modeling for speech units and text. Specifically, we compare different speech tokenizers to transform continuous speech signals into discrete units and use different methods to construct mixed speech-text data. We introduce automatic metrics to evaluate how well the joint LM mixes speech and text. We also fine-tune the LM on downstream spoken language understanding (SLU) tasks with different modalities (speech or text) and test its performance to assess the model's learning of shared representations. Our results show that by mixing speech units and text with our proposed mixing techniques, the joint LM improves over a speech-only baseline on SLU tasks and shows zero-shot cross-modal transferability.

Key facts. Cited by 28 papers in the map (data through 14 Oct 2026). Builds on 41 of its 47 references that are in the map. First posted on arXiv 12 Oct 2023.

Builds on

41 of 47 references.

References · 41

  1. 2017Attention Is All You Needcited by 60,361
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  3. 2014Adam: A Method for Stochastic Optimizationcited by 37,240
  4. 2020Language Models are Few-Shot Learnerscited by 32,419
  5. 2019Unsupervised Cross-lingual Representation Learning at Scalecited by 3,935
  6. 2020wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representationscited by 3,905
  7. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,512
  8. 2019The Curious Case of Neural Text Degenerationcited by 2,916
  9. 2021HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Unitscited by 2,342
  10. 2018SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processingcited by 1,989
  11. 2020Conformer: Convolution-augmented Transformer for Speech Recognitioncited by 1,863
  12. 2021WavLM: Large-Scale Self-Supervised Pre-Training for Full Stack Speech Processingcited by 1,657
29 more
  1. 2019fairseq: A Fast, Extensible Toolkit for Sequence Modelingcited by 1,488
  2. 2019Common Voice: A Massively-Multilingual Speech Corpuscited by 1,384
  3. 2023Neural Codec Language Models are Zero-Shot Text to Speech Synthesizerscited by 752
  4. 2021GPT Understands, Toocited by 716
  5. 2021SUPERB: Speech processing Universal PERformance Benchmarkcited by 666
  6. 2020MLS: A Large-Scale Multilingual Dataset for Speech Researchcited by 476
  7. 2021VoxPopuli: A Large-Scale Multilingual Speech Corpus for Representation Learning, Semi-Supervised Learning and Interpretationcited by 463
  8. 2021Generative Spoken Language Modeling from Raw Audiocited by 340
  9. 2016Using the Output Embedding to Improve Language Modelscited by 272
  10. 2021Speech Resynthesis from Discrete Disentangled Self-Supervised Representationscited by 243
  11. 2021SpeechT5: Unified-Modal Encoder-Decoder Pre-Training for Spoken Language Processingcited by 180
  12. 2022Comparative layer-wise analysis of self-supervised speech modelscited by 158
  13. 2021Direct speech-to-speech translation with discrete unitscited by 149
  14. 2023Scaling Laws for Generative Mixed-Modal Language Modelscited by 134
  15. 2021Textless Speech-to-Speech Translation on Real Datacited by 121
  16. 2021Text-Free Prosody-Aware Generative Spoken Language Modelingcited by 119
  17. 2022mSLAM: Massively multilingual joint pre-training for speech and textcited by 108
  18. 2023EXPRESSO: A Benchmark and Analysis of Discrete Expressive Speech Resynthesiscited by 98
  19. 2018Language GANs Falling Shortcited by 97
  20. 2022MAESTRO: Matched Speech Text Representations through Modality Matchingcited by 79
  21. 2021SLUE: New Benchmark Tasks for Spoken Language Understanding Evaluation on Natural Speechcited by 72
  22. 2021SLAM: A Unified Encoder for Speech and Language Modeling via Speech-Text Joint Pre-Trainingcited by 69
  23. 2022Unified Speech-Text Pre-training for Speech Translation and Recognitioncited by 64
  24. 2022SpeechLM: Enhanced Speech Pre-Training with Unpaired Textual Datacited by 49
  25. 2022u-HuBERT: Unified Mixed-Modal Speech Pretraining And Zero-Shot Transfer to Unlabeled Modalitycited by 39
  26. 2022Wav2Seq: Pre-training Speech-to-Text Encoder-Decoder Models Using Pseudo Languagescited by 26
  27. 2022Speech-to-Speech Translation For A Real-world Unwritten Languagecited by 25
  28. 2022ReVISE: Self-Supervised Speech Resynthesis with Visual Input for Universal and Generalized Speech Enhancementcited by 8
  29. 2022Mu2SLAM: Multitask, Multilingual Speech and Language Modelscited by 7

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 17; 2025: 5; 2026: 6.

28 papers.

Sort:

By area:

  1. 2024Spirit LM: Interleaved Spoken and Written Language Modelcited by 126
  2. 2025On The Landscape of Spoken Language Models: A Comprehensive Surveycited by 118
  3. 2024Recent Advances in Speech Language Models: A Surveycited by 87
  4. 2024Beyond Turn-Based Interfaces: Synchronous LLMs as Full-Duplex Dialogue Agentscited by 72
  5. 2024mHuBERT-147: A Compact Multilingual HuBERT Modelcited by 66
  6. 2024Advancing Large Language Models to Capture Varied Speaking Styles and Respond Properly in Spoken Conversationscited by 45
  7. 2024SyllableLM: Learning Coarse Semantic Units for Speech Language Modelscited by 24
  8. 2024Align-SLM: Textless Spoken Language Models with Reinforcement Learning from AI Feedbackcited by 23
  9. 2024PSLM: Parallel Generation of Text and Speech with LLMs for Low-Latency Spoken Dialogue Systemscited by 14
  10. 2024DiscreteSLU: A Large Language Model with Self-Supervised Discrete Speech Units for Spoken Language Understandingcited by 10
  11. 2024On the Evaluation of Speech Foundation Models for Spoken Language Understandingcited by 9
  12. 2024CogniVoice: Multimodal and Multilingual Fusion Networks for Mild Cognitive Impairment Assessment from Spontaneous Speechcited by 8
16 more
  1. 2024Improving Spoken Language Modeling with Phoneme Classification: A Simple Fine-tuning Approachcited by 8
  2. 2025From TOWER to SPIRE: Adding the Speech Modality to a Translation-Specialist LLMcited by 8
  3. 2025Flow-SLM: Joint Learning of Linguistic and Acoustic Information for Spoken Language Modelingcited by 7
  4. 2024Get Large Language Models Ready to Speak: A Late-fusion Approach for Speech Generationcited by 5
  5. 2024Speech Recognition Rescoring with Large Speech-Text Foundation Modelscited by 2
  6. 2025Efficient Interleaved Speech Modeling through Knowledge Distillationcited by 2
  7. 2024Direct Punjabi to English speech translation using discrete unitscited by 1
  8. 2024Compact Speech Translation Models via Discrete Speech Units Pretrainingcited by 1
  9. 2026LoASR-Bench: Evaluating Large Speech Language Models on Low-Resource Automatic Speech Recognition Across Language Familiescited by 1
  10. 2026Lexical Tone is Hard to Quantize: Probing Discrete Speech Units in Mandarin and Yor`ub'acited by 1
  11. 2026How Tokenization Limits Phonological Knowledge Representation in Language Models and How to Improve Themcited by 1
  12. 2026Quantifying the Generation Modality Gap in Speech-Text Language Modelscited by 1
  13. 2024Enhancing Code-switched Text-to-Speech Synthesis Capability in Large Language Models with only Monolingual Corporacited by 0
  14. 2025Cross-Lingual Interleaving for Speech Language Modelscited by 0
  15. 2026SPAR-K: Scheduled Periodic Alternating Early Exit for Spoken Language Modelscited by 0
  16. 2026Distilling Text LMs into Speech-to-Speech LMs by Matching Internal Text Predictionscited by 0