mlsanityData to 7 Oct 2026

SpeechLM: Enhanced Speech Pre-Training with Unpaired Textual Data

arXiv 2209.15329 ↗First posted 30 Sep 2022cs.CL cs.AI eess.AScited by 49

Not on the map. Why?

Abstract (arXiv)

How to boost speech pre-training with textual data is an unsolved problem due to the fact that speech and text are very different modalities with distinct characteristics. In this paper, we propose a cross-modal Speech and Language Model (SpeechLM) to explicitly align speech and text pre-training with a pre-defined unified discrete representation. Specifically, we introduce two alternative discrete tokenizers to bridge the speech and text modalities, including phoneme-unit and hidden-unit tokenizers, which can be trained using a small amount of paired speech-text data. Based on the trained tokenizers, we convert the unlabeled speech and text data into tokens of phoneme units or hidden units. The pre-training objective is designed to unify the speech and the text into the same discrete semantic space with a unified Transformer network. We evaluate SpeechLM on various spoken language processing tasks including speech recognition, speech translation, and universal representation evaluation framework SUPERB, demonstrating significant improvements on content-related tasks. Code and models are available at https://aka.ms/SpeechLM.

Key facts. Cited by 49 papers in the map (data through 7 Oct 2026). Builds on 36 of its 54 references that are in the map. First posted on arXiv 30 Sep 2022.

Builds on

36 of 54 references.

References · 36

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  4. 2020wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representationscited by 3,853
  5. 2021HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Unitscited by 2,301
  6. 2021WavLM: Large-Scale Self-Supervised Pre-Training for Full Stack Speech Processingcited by 1,609
  7. 2018Self-Attention with Relative Position Representationscited by 1,112
  8. 2019SpanBERT: Improving Pre-training by Representing and Predicting Spanscited by 832
  9. 2022data2vec: A General Framework for Self-supervised Learning in Speech, Vision and Languagecited by 718
  10. 2019Unified Language Model Pre-training for Natural Language Understanding and Generationcited by 668
  11. 2021SUPERB: Speech processing Universal PERformance Benchmarkcited by 652
  12. 2019Libri-Light: A Benchmark for ASR with Limited or No Supervisioncited by 462
24 more
  1. 2021W2v-BERT: Combining Contrastive Learning and Masked Language Modeling for Self-Supervised Speech Pre-Trainingcited by 340
  2. 2019vq-wav2vec: Self-Supervised Learning of Discrete Speech Representationscited by 335
  3. 2019Mockingjay: Unsupervised Speech Representation Learning with Deep Bidirectional Transformer Encoderscited by 196
  4. 2020TERA: Self-Supervised Learning of Transformer Encoder Representation for Speechcited by 186
  5. 2019An Unsupervised Autoregressive Model for Speech Representation Learningcited by 185
  6. 2021SpeechT5: Unified-Modal Encoder-Decoder Pre-Training for Spoken Language Processingcited by 177
  7. 2021Unsupervised Speech Recognitioncited by 147
  8. 2020Multi-task self-supervised learning for Robust Speech Recognitioncited by 127
  9. 2022mSLAM: Massively multilingual joint pre-training for speech and textcited by 108
  10. 2020Unsupervised pretraining transfers well across languagescited by 100
  11. 2022MAESTRO: Matched Speech Text Representations through Modality Matchingcited by 79
  12. 2020DeCoAR 2.0: Deep Contextualized Acoustic Representations with Vector Quantizationcited by 77
  13. 2021UniSpeech: Unified Speech Representation Learning with Labeled and Unlabeled Datacited by 76
  14. 2020Vector-Quantized Autoregressive Predictive Codingcited by 69
  15. 2021SLAM: A Unified Encoder for Speech and Language Modeling via Speech-Text Joint Pre-Trainingcited by 69
  16. 2022Unified Speech-Text Pre-training for Speech Translation and Recognitioncited by 64
  17. 2020Non-Autoregressive Predictive Coding for Learning Speech Representations from Local Dependenciescited by 47
  18. 2021Large-Scale Self- and Semi-Supervised Learning for Speech Translationcited by 36
  19. 2021Speech-language Pre-training for End-to-end Spoken Language Understandingcited by 27
  20. 2020ST-BERT: Cross-modal Language Model Pre-training For End-to-end Spoken Language Understandingcited by 19
  21. 2021Self-Supervised Learning for speech recognition with Intermediate layer supervisioncited by 15
  22. 2022Supervision-Guided Codebooks for Masked Prediction in Speech Pre-trainingcited by 14
  23. 2021Optimizing Alignment of Speech and Language Latent Spaces for End-to-End Speech Recognition and Understandingcited by 11
  24. 2021An Investigation of End-to-End Models for Robust Speech Recognitioncited by 9

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 6; 2023: 17; 2024: 13; 2025: 7; 2026: 6.

49 papers.

Sort:

By area:

  1. 2023Google USM: Scaling Automatic Speech Recognition Beyond 100 Languagescited by 228
  2. 2023Speak Foreign Languages with Your Own Voice: Cross-Lingual Neural Codec Language Modelingcited by 166
  3. 2023ONE-PEACE: Exploring One General Representation Model Toward Unlimited Modalitiescited by 117
  4. 2023Spoken Question Answering and Speech Continuation Using Spectrogram-Powered LLMcited by 92
  5. 2023SALM: Speech-augmented Language Model with In-context Learning for Speech Recognition and Translationcited by 75
  6. 2024Codec Does Matter: Exploring the Semantic Shortcoming of Codec for Audio Language Modelcited by 70
  7. 2022SpeechUT: Bridging Speech and Text with Hidden-Unit for Encoder-Decoder Based Speech-Text Pre-trainingcited by 40
  8. 2022VATLM: Visual-Audio-Text Pre-Training with Unified Masked Prediction for Speech Representation Learningcited by 34
  9. 2023Toward Joint Language Modeling for Speech Units and Textcited by 27
  10. 2024More Than Routing: Joint GPS and Route Modeling for Refine Trajectory Representation Learningcited by 20
  11. 2024HoME: Hierarchy of Multi-Gate Experts for Multi-Task Learning at Kuaishoucited by 16
  12. 2025Watermarking Autoregressive Image Generationcited by 15
37 more
  1. 2023Translatotron 3: Speech to Speech Translation with Monolingual Datacited by 12
  2. 2022MT4SSL: Boosting Self-Supervised Speech Representation Learning by Integrating Multiple Targetscited by 8
  3. 2023GSQA: An End-to-End Model for Generative Spoken Question Answeringcited by 8
  4. 2022Mu2SLAM: Multitask, Multilingual Speech and Language Modelscited by 7
  5. 2024DM-Codec: Distilling Multimodal Representations for Speech Tokenizationcited by 5
  6. 2023Conversational Speech Recognition by Learning Audio-textual Cross-modal Contextual Representationcited by 4
  7. 2023A Comprehensive Study on the Effectiveness of ASR Representations for Noise-Robust Speech Emotion Recognitioncited by 4
  8. 2024Exploring the Benefits of Tokenization of Discrete Acoustic Unitscited by 4
  9. 2024Seamless Language Expansion: Enhancing Multilingual Mastery in Self-Supervised Modelscited by 4
  10. 2025Spoken in Jest, Detected in Earnest: A Systematic Review of Sarcasm Recognition -- Multimodal Fusion, Challenges, and Future Prospectscited by 4
  11. 2022token2vec: A Joint Self-Supervised Pre-training Framework Using Unpaired Speech and Textcited by 3
  12. 2022Introducing Semantics into Speech Encoderscited by 3
  13. 2024An efficient text augmentation approach for contextualized Mandarin speech recognitioncited by 2
  14. 2024CJST: CTC Compressor based Joint Speech and Text Training for Decoder-Only ASRcited by 2
  15. 2025Speech Translation Refinement using Large Language Modelscited by 2
  16. 2025UDDETTS: Unifying Discrete and Dimensional Emotions for Controllable Emotional Text-to-Speechcited by 2
  17. 2025NaturalVoices: A Large-Scale, Spontaneous and Emotional Podcast Dataset for Voice Conversioncited by 2
  18. 2023Speech Translation with Foundation Models and Optimal Transport: UPC at IWSLT23cited by 1
  19. 2023EM-Network: Oracle Guided Self-distillation for Sequence Learningcited by 1
  20. 2023Robust Automatic Speech Recognition via WavAugment Guided Phoneme Adversarial Trainingcited by 1
  21. 2023UniBriVL: Robust Universal Representation and Generation of Audio Driven Diffusion Modelscited by 1
  22. 2023Few-Shot Spoken Language Understanding via Joint Speech-Text Modelscited by 1
  23. 2024Deep Learning Models in Speech Recognition: Measuring GPU Energy Consumption, Impact of Noise and Model Quantization for Edge Deploymentcited by 1
  24. 2024Towards Unsupervised Speech Recognition Without Pronunciation Modelscited by 1
  25. 2024Text-to-Speech for Unseen Speakers via Low-Complexity Discrete Unit-Based Frame Selectioncited by 1
  26. 2024CTC-GMM: CTC guided modality matching for fast and accurate streaming speech translationcited by 1
  27. 2025AsyncSwitch: Asynchronous Text-Speech Adaptation for Code-Switched ASRcited by 1
  28. 2025Vox-Evaluator: Enhancing Stability and Fidelity for Zero-shot TTS with A Multi-Level Evaluatorcited by 1
  29. 2026LoASR-Bench: Evaluating Large Speech Language Models on Low-Resource Automatic Speech Recognition Across Language Familiescited by 1
  30. 2026From Physics to Representation: Audio Learning with Synthetic Pre-training via Procedural Generationcited by 1
  31. 2023Typing on Any Surface: A Deep Learning-based Method for Real-Time Keystroke Detection in Augmented Realitycited by 0
  32. 2023Improving End-to-End Speech Processing by Efficient Text Data Utilization with Latent Synthesiscited by 0
  33. 2024Bridging the Gap between Text, Audio, Image, and Any Sequence: A Novel Approach using Gloss-based Annotationcited by 0
  34. 2026Evaluating Time Awareness and Cross-modal Active Perception of Large Models via 4D Escape Room Taskcited by 0
  35. 2026Which Speech Representation Better Matches Text-Native Reasoning? A Study of Speech-Text Alignment on Frame Rate and Representationcited by 0
  36. 2026Rethinking Speech-LLM Integration for ASR: Effective Joint Speech-Text Training by Interleavingcited by 0
  37. 2026Automatic Lyric Transcription for Greek Songs: Scaling and Task Composition Effects in Whisper Adaptationcited by 0