mlsanityData to 14 Oct 2026

SpeechUT: Bridging Speech and Text with Hidden-Unit for Encoder-Decoder Based Speech-Text Pre-training

arXiv 2210.03730 ↗First posted 7 Oct 2022cs.CL eess.AScited by 40

Not on the map. Why?

Abstract (arXiv)

The rapid development of single-modal pre-training has prompted researchers to pay more attention to cross-modal pre-training methods. In this paper, we propose a unified-modal speech-unit-text pre-training model, SpeechUT, to connect the representations of a speech encoder and a text decoder with a shared unit encoder. Leveraging hidden-unit as an interface to align speech and text, we can decompose the speech-to-text model into a speech-to-unit model and a unit-to-text model, which can be jointly pre-trained with unpaired speech and text data respectively. Our proposed SpeechUT is fine-tuned and evaluated on automatic speech recognition (ASR) and speech translation (ST) tasks. Experimental results show that SpeechUT gets substantial improvements over strong baselines, and achieves state-of-the-art performance on both the LibriSpeech ASR and MuST-C ST tasks. To better understand the proposed SpeechUT, detailed analyses are conducted. The code and pre-trained models are available at https://aka.ms/SpeechUT.

Key facts. Cited by 40 papers in the map (data through 14 Oct 2026). Builds on 39 of its 54 references that are in the map. First posted on arXiv 7 Oct 2022.

Builds on

39 of 54 references.

References · 39

  1. 2017Attention Is All You Needcited by 60,361
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  3. 2019BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehensioncited by 5,893
  4. 2017Neural Discrete Representation Learningcited by 4,712
  5. 2020wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representationscited by 3,905
  6. 2019XLNet: Generalized Autoregressive Pretraining for Language Understandingcited by 2,811
  7. 2021HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Unitscited by 2,342
  8. 2018SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processingcited by 1,989
  9. 2021WavLM: Large-Scale Self-Supervised Pre-Training for Full Stack Speech Processingcited by 1,657
  10. 2019fairseq: A Fast, Extensible Toolkit for Sequence Modelingcited by 1,488
  11. 2018Self-Attention with Relative Position Representationscited by 1,120
  12. 2022data2vec: A General Framework for Self-supervised Learning in Speech, Vision and Languagecited by 727
27 more
  1. 2019Unified Language Model Pre-training for Natural Language Understanding and Generationcited by 670
  2. 2019Libri-Light: A Benchmark for ASR with Limited or No Supervisioncited by 467
  3. 2021W2v-BERT: Combining Contrastive Learning and Masked Language Modeling for Self-Supervised Speech Pre-Trainingcited by 347
  4. 2019vq-wav2vec: Self-Supervised Learning of Discrete Speech Representationscited by 340
  5. 2020Pushing the Limits of Semi-Supervised Learning for Automatic Speech Recognitioncited by 187
  6. 2021SpeechT5: Unified-Modal Encoder-Decoder Pre-Training for Spoken Language Processingcited by 180
  7. 2015On Using Monolingual Corpora in Neural Machine Translationcited by 150
  8. 2021Direct speech-to-speech translation with discrete unitscited by 149
  9. 2021Textless Speech-to-Speech Translation on Real Datacited by 121
  10. 2019End-to-end ASR: from Supervised to Semi-Supervised Learning with Modern Architecturescited by 120
  11. 2022mSLAM: Massively multilingual joint pre-training for speech and textcited by 108
  12. 2020Self-training and Pre-training are Complementary for Speech Recognitioncited by 99
  13. 2022MAESTRO: Matched Speech Text Representations through Modality Matchingcited by 79
  14. 2021SLAM: A Unified Encoder for Speech and Language Modeling via Speech-Text Joint Pre-Trainingcited by 69
  15. 2022Unified Speech-Text Pre-training for Speech Translation and Recognitioncited by 64
  16. 2022Cross-modal Contrastive Learning for Speech Translationcited by 61
  17. 2022STEMM: Self-learning with Speech-text Manifold Mixup for Speech Translationcited by 57
  18. 2022SpeechLM: Enhanced Speech Pre-Training with Unpaired Textual Datacited by 49
  19. 2021Stacked Acoustic-and-Textual Encoding: Integrating the Pre-trained Models into Speech Translation Encoderscited by 48
  20. 2021Fused Acoustic and Text Encoding for Multimodal Bilingual Pretraining and Speech Translationcited by 47
  21. 2021Speech-language Pre-training for End-to-end Spoken Language Understandingcited by 27
  22. 2022Wav2Seq: Pre-training Speech-to-Text Encoder-Decoder Models Using Pseudo Languagescited by 26
  23. 2020ST-BERT: Cross-modal Language Model Pre-training For End-to-end Spoken Language Understandingcited by 19
  24. 2021Self-Supervised Learning for speech recognition with Intermediate layer supervisioncited by 15
  25. 2022Supervision-Guided Codebooks for Masked Prediction in Speech Pre-trainingcited by 14
  26. 2022Pre-Training Transformer Decoder for End-to-End ASR Model with Unpaired Speech Datacited by 12
  27. 2021Optimizing Alignment of Speech and Language Latent Spaces for End-to-End Speech Recognition and Understandingcited by 11

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2022: 8; 2023: 19; 2024: 8; 2025: 3; 2026: 2.

40 papers.

Sort:

  1. 2023Google USM: Scaling Automatic Speech Recognition Beyond 100 Languagescited by 231
  2. 2023Speak Foreign Languages with Your Own Voice: Cross-Lingual Neural Codec Language Modelingcited by 167
  3. 2023ONE-PEACE: Exploring One General Representation Model Toward Unlimited Modalitiescited by 118
  4. 2022VATLM: Visual-Audio-Text Pre-Training with Unified Masked Prediction for Speech Representation Learningcited by 37
  5. 2024Where Visual Speech Meets Language: VSP-LLM Framework for Efficient and Context-Aware Visual Speech Processingcited by 30
  6. 2023AV-data2vec: Self-supervised Learning of Audio-Visual Speech Representations with Contextualized Target Representationscited by 27
  7. 2023CMOT: Cross-modal Mixup via Optimal Transport for Speech Translationcited by 22
  8. 2022WACO: Word-Aligned Contrastive Learning for Speech Translationcited by 21
  9. 2023DUB: Discrete Unit Back-translation for Speech Translationcited by 18
  10. 2023Understanding and Bridging the Modality Gap for Speech Translationcited by 15
  11. 2023CopyNE: Better Contextual ASR by Copying Named Entitiescited by 14
  12. 2024Pushing the Limits of Zero-shot End-to-End Speech Translationcited by 14
28 more
  1. 2023Textless Unit-to-Unit training for Many-to-Many Multilingual Speech-to-Speech Translationcited by 11
  2. 2024TransVIP: Speech to Speech Translation System with Voice and Isochrony Preservationcited by 10
  3. 2022Joint Pre-Training with Speech and Bilingual Text for Direct Speech to Speech Translationcited by 9
  4. 2022MMSpeech: Multi-modal Multi-task Encoder-Decoder Pre-training for Speech Recognitioncited by 9
  5. 2022Improving End-to-end Speech Translation by Leveraging Auxiliary Speech and Text Datacited by 9
  6. 2023Back Translation for Speech-to-text Translation Without Transcriptscited by 9
  7. 2023ComSL: A Composite Speech-Language Model for End-to-End Speech-to-Text Translationcited by 9
  8. 2023Tuning Large language model for End-to-end Speech Translationcited by 9
  9. 2022Mu2SLAM: Multitask, Multilingual Speech and Language Modelscited by 7
  10. 2023Improving Speech Translation by Cross-Modal Multi-Grained Contrastive Learningcited by 7
  11. 2022SegAugment: Maximizing the Utility of Speech Translation Data with Segmentation-based Augmentationscited by 6
  12. 2023Modality Adaption or Regularization? A Case Study on End-to-End Speech Translationcited by 6
  13. 2023An Empirical Study of Consistency Regularization for End-to-End Speech-to-Text Translationcited by 6
  14. 2023Rethinking and Improving Multi-task Learning for End-to-end Speech Translationcited by 6
  15. 2023Soft Alignment of Modality Space for End-to-end Speech Translationcited by 4
  16. 2022AdaTranS: Adapting with Boundary-based Shrinking for End-to-End Speech Translationcited by 3
  17. 2024CLASP: Contrastive Language-Speech Pretraining for Multilingual Multimodal Information Retrievalcited by 2
  18. 2025Speech Translation Refinement using Large Language Modelscited by 2
  19. 2023Few-Shot Spoken Language Understanding via Joint Speech-Text Modelscited by 1
  20. 2024UniEnc-CASSNAT: An Encoder-only Non-autoregressive ASR for Speech SSL Modelscited by 1
  21. 2024Compact Speech Translation Models via Discrete Speech Units Pretrainingcited by 1
  22. 2024SBAAM! Eliminating Transcript Dependency in Automatic Subtitlingcited by 1
  23. 2024Towards Unsupervised Speech Recognition Without Pronunciation Modelscited by 1
  24. 2025Enhanced Hybrid Transducer and Attention Encoder Decoder with Text Datacited by 1
  25. 2026LoASR-Bench: Evaluating Large Speech Language Models on Low-Resource Automatic Speech Recognition Across Language Familiescited by 1
  26. 2023Improving End-to-End Speech Processing by Efficient Text Data Utilization with Latent Synthesiscited by 0
  27. 2025Interpretable Style Takagi-Sugeno-Kang Fuzzy Clusteringcited by 0
  28. 2026Text-Utilization for Encoder-dominated Speech Recognition Modelscited by 0