mlsanityData to 7 Oct 2026

Unified Speech-Text Pre-training for Speech Translation and Recognition

arXiv 2204.05409 ↗First posted 11 Apr 2022cs.CLcited by 64

Not on the map. Why?

Abstract (arXiv)

We describe a method to jointly pre-train speech and text in an encoder-decoder modeling framework for speech translation and recognition. The proposed method incorporates four self-supervised and supervised subtasks for cross modality learning. A self-supervised speech subtask leverages unlabelled speech data, and a (self-)supervised text to text subtask makes use of abundant text training data. Two auxiliary supervised speech tasks are included to unify speech and text modeling space. Our contribution lies in integrating linguistic information from the text corpus into the speech pre-training. Detailed analysis reveals learning interference among subtasks. Two pre-training configurations for speech translation and recognition, respectively, are presented to alleviate subtask interference. Our experiments show the proposed method can effectively fuse speech and text information into one model. It achieves between 1.7 and 2.3 BLEU improvement above the state of the art on the MuST-C speech translation dataset and comparable WERs to wav2vec 2.0 on the Librispeech speech recognition task.

Key facts. Cited by 64 papers in the map (data through 7 Oct 2026). Builds on 37 of its 58 references that are in the map. First posted on arXiv 11 Apr 2022.

Builds on

37 of 58 references.

References · 37

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  3. 2018Representation Learning with Contrastive Predictive Codingcited by 7,662
  4. 2019BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehensioncited by 5,874
  5. 2016Layer Normalizationcited by 5,579
  6. 2020wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representationscited by 3,853
  7. 2018Deep contextualized word representationscited by 2,285
  8. 2018SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processingcited by 1,973
  9. 2018A Call for Clarity in Reporting BLEU Scorescited by 1,790
  10. 2019SpecAugment: A Simple Data Augmentation Method for Automatic Speech Recognitioncited by 1,355
  11. 2020Multilingual Denoising Pre-training for Neural Machine Translationcited by 1,028
  12. 2020On Layer Normalization in the Transformer Architecturecited by 906
25 more
  1. 2019Libri-Light: A Benchmark for ASR with Limited or No Supervisioncited by 462
  2. 2020Rethinking Pre-training and Self-trainingcited by 352
  3. 2021W2v-BERT: Combining Contrastive Learning and Masked Language Modeling for Self-Supervised Speech Pre-Trainingcited by 340
  4. 2019vq-wav2vec: Self-Supervised Learning of Discrete Speech Representationscited by 335
  5. 2020Pushing the Limits of Semi-Supervised Learning for Automatic Speech Recognitioncited by 185
  6. 2021SpeechT5: Unified-Modal Encoder-Decoder Pre-Training for Spoken Language Processingcited by 177
  7. 2019Self-Training for End-to-End Speech Recognitioncited by 128
  8. 2017Sequence-to-Sequence Models Can Directly Translate Foreign Speechcited by 121
  9. 2020ESPnet-ST: All-in-One Speech Translation Toolkitcited by 82
  10. 2020Iterative Pseudo-Labeling for Speech Recognitioncited by 81
  11. 2021SLAM: A Unified Encoder for Speech and Language Modeling via Speech-Text Joint Pre-Trainingcited by 69
  12. 2018Leveraging Weakly Supervised Data to Improve End-to-End Speech-to-Text Translationcited by 57
  13. 2018Tied Multitask Learning for Neural Speech Translationcited by 54
  14. 2021Improving Speech Translation by Understanding and Learning from the Auxiliary Text Translation Taskcited by 54
  15. 2020A General Multi-Task Learning Framework to Leverage Text Data for Speech to Text Taskscited by 51
  16. 2021Fused Acoustic and Text Encoding for Multimodal Bilingual Pretraining and Speech Translationcited by 47
  17. 2021End-to-end Speech Translation via Cross-modal Progressive Trainingcited by 47
  18. 2020Self-Training for End-to-End Speech Translationcited by 36
  19. 2021Large-Scale Self- and Semi-Supervised Learning for Speech Translationcited by 36
  20. 2018Unsupervised Cross-Modal Alignment of Speech and Text Embedding Spacescited by 34
  21. 2021Injecting Text in Self-Supervised Speech Pretrainingcited by 24
  22. 2020Improved Speech Representations with Multi-Target Autoregressive Predictive Codingcited by 23
  23. 2018Learning pronunciation from a foreign language in speech synthesis networkscited by 12
  24. 2021Contrastive Semi-supervised Learning for ASRcited by 11
  25. 2021UniSpeech at scale: An Empirical Study of Pre-training Method on Large-Scale Speech Recognition Datasetcited by 7

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 18; 2023: 27; 2024: 9; 2025: 7; 2026: 3.

64 papers.

Sort:

By area:

  1. 2023End-to-End Speech Recognition: A Surveycited by 127
  2. 2023ONE-PEACE: Exploring One General Representation Model Toward Unlimited Modalitiescited by 117
  3. 2022Building Machine Translation Systems for the Next Thousand Languagescited by 69
  4. 2022Cross-modal Contrastive Learning for Speech Translationcited by 61
  5. 2022UnitY: Two-pass Direct Speech-to-speech Translation with Discrete Unitscited by 60
  6. 2022SpeechLM: Enhanced Speech Pre-Training with Unpaired Textual Datacited by 49
  7. 2022SpeechUT: Bridging Speech and Text with Hidden-Unit for Encoder-Decoder Based Speech-Text Pre-trainingcited by 40
  8. 2023SLTUNET: A Simple Unified Model for Sign Language Translationcited by 35
  9. 2022VATLM: Visual-Audio-Text Pre-Training with Unified Masked Prediction for Speech Representation Learningcited by 34
  10. 2023Toward Joint Language Modeling for Speech Units and Textcited by 27
  11. 2023Pre-training for Speech Translation: CTC Meets Optimal Transportcited by 25
  12. 2022JOIST: A Joint Speech and Text Streaming Model For ASRcited by 24
38 more
  1. 2022WACO: Word-Aligned Contrastive Learning for Speech Translationcited by 21
  2. 2023Hybrid Transducer and Attention based Encoder-Decoder Modeling for Speech-to-Text Taskscited by 20
  3. 2022Virtuoso: Massive Multilingual Speech-Text Joint Semi-Supervised Learning for Text-To-Speechcited by 15
  4. 2023Understanding and Bridging the Modality Gap for Speech Translationcited by 15
  5. 2023Recent Advances in Direct Speech-to-text Translationcited by 15
  6. 2024Pushing the Limits of Zero-shot End-to-End Speech Translationcited by 14
  7. 2023Speech-Text Dialog Pre-training for Spoken Dialog Understanding with Explicit Cross-Modal Alignmentcited by 13
  8. 2023CopyNE: Better Contextual ASR by Copying Named Entitiescited by 13
  9. 2023Translatotron 3: Speech to Speech Translation with Monolingual Datacited by 12
  10. 2022LAMASSU: Streaming Language-Agnostic Multilingual Speech Recognition and Translation Using Neural Transducerscited by 10
  11. 2022Speech Aware Dialog System Technology Challenge (DSTC11)cited by 10
  12. 2023Massively Multilingual Shallow Fusion with Large Language Modelscited by 10
  13. 2023Code-Switching Text Generation and Injection in Mandarin-English ASRcited by 10
  14. 2022MMSpeech: Multi-modal Multi-task Encoder-Decoder Pre-training for Speech Recognitioncited by 9
  15. 2022Improving End-to-end Speech Translation by Leveraging Auxiliary Speech and Text Datacited by 9
  16. 2023JEIT: Joint End-to-End Model and Internal Language Model Training for Speech Recognitioncited by 9
  17. 2023Back Translation for Speech-to-text Translation Without Transcriptscited by 9
  18. 2023ComSL: A Composite Speech-Language Model for End-to-End Speech-to-Text Translationcited by 9
  19. 2023Tuning Large language model for End-to-end Speech Translationcited by 9
  20. 2023End-to-End Speech-to-Text Translation: A Surveycited by 9
  21. 2026MOSS-Audio Technical Reportcited by 9
  22. 2023Incremental Blockwise Beam Search for Simultaneous Speech Translation with Controllable Quality-Latency Tradeoffcited by 8
  23. 2022Mu2SLAM: Multitask, Multilingual Speech and Language Modelscited by 7
  24. 2023Improving Speech Translation by Cross-Modal Multi-Grained Contrastive Learningcited by 7
  25. 2022SegAugment: Maximizing the Utility of Speech Translation Data with Segmentation-based Augmentationscited by 6
  26. 2023Adapting Offline Speech Translation Models for Streaming with Future-Aware Distillation and Inferencecited by 6
  27. 2023An Empirical Study of Consistency Regularization for End-to-End Speech-to-Text Translationcited by 6
  28. 2022Joint Audio/Text Training for Transformer Rescorer of Streaming Speech Recognitioncited by 5
  29. 2023Text-only Domain Adaptation using Unified Speech-Text Representation in Transducercited by 5
  30. 2022Named Entity Detection and Injection for Direct Speech Translationcited by 4
  31. 2024CoSTA: Code-Switched Speech Translation using Aligned Speech-Text Interleavingcited by 4
  32. 2024EMMeTT: Efficient Multimodal Machine Translation Trainingcited by 4
  33. 2024Effective Text Adaptation for LLM-based ASR through Soft Prompt Fine-Tuningcited by 4
  34. 2022AdaTranS: Adapting with Boundary-based Shrinking for End-to-End Speech Translationcited by 3
  35. 2023Improving speech translation by fusing speech and textcited by 3
  36. 2023Cross-Modal Multi-Tasking for Speech-to-Text Translation via Hard Parameter Sharingcited by 3
  37. 2024Text Injection for Neural Contextual Biasingcited by 3
  38. 2024CA-SSLR: Condition-Aware Self-Supervised Learning Representation for Generalized Speech Processingcited by 2

50 most cited of 64 All 64 (JSON)