mlsanityData to 14 Oct 2026

SpeechT5: Unified-Modal Encoder-Decoder Pre-Training for Spoken Language Processing

arXiv 2110.07205 ↗First posted 14 Oct 2021eess.AS cs.CL cs.LGcited by 180

Not on the map. Why?

Abstract (arXiv)

Motivated by the success of T5 (Text-To-Text Transfer Transformer) in pre-trained natural language processing models, we propose a unified-modal SpeechT5 framework that explores the encoder-decoder pre-training for self-supervised speech/text representation learning. The SpeechT5 framework consists of a shared encoder-decoder network and six modal-specific (speech/text) pre/post-nets. After preprocessing the input speech/text through the pre-nets, the shared encoder-decoder network models the sequence-to-sequence transformation, and then the post-nets generate the output in the speech/text modality based on the output of the decoder. Leveraging large-scale unlabeled speech and text data, we pre-train SpeechT5 to learn a unified-modal representation, hoping to improve the modeling capability for both speech and text. To align the textual and speech information into this unified semantic space, we propose a cross-modal vector quantization approach that randomly mixes up speech/text states with latent units as the interface between encoder and decoder. Extensive evaluations show the superiority of the proposed SpeechT5 framework on a wide variety of spoken language processing tasks, including automatic speech recognition, speech synthesis, speech translation, voice conversion, speech enhancement, and speaker identification. We release our code and model at https://github.com/microsoft/SpeechT5.

Key facts. Cited by 180 papers in the map (data through 14 Oct 2026). Builds on 59 of its 79 references that are in the map. First posted on arXiv 14 Oct 2021.

Builds on

59 of 79 references.

References · 59

  1. 2017Attention Is All You Needcited by 60,361
  2. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  3. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  4. 2014Adam: A Method for Stochastic Optimizationcited by 37,240
  5. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  6. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,372
  7. 2019BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehensioncited by 5,893
  8. 2016Layer Normalizationcited by 5,669
  9. 2017Neural Discrete Representation Learningcited by 4,712
  10. 2020wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representationscited by 3,905
  11. 2019XLNet: Generalized Autoregressive Pretraining for Language Understandingcited by 2,811
  12. 2021BEiT: BERT Pre-Training of Image Transformerscited by 2,375
38 more
  1. 2021HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Unitscited by 2,342
  2. 2018Deep contextualized word representationscited by 2,290
  3. 2019fairseq: A Fast, Extensible Toolkit for Sequence Modelingcited by 1,488
  4. 2020HiFi-GAN: Generative Adversarial Networks for Efficient and High Fidelity Speech Synthesiscited by 1,154
  5. 2018Self-Attention with Relative Position Representationscited by 1,120
  6. 2019Cross-lingual Language Model Pretrainingcited by 1,090
  7. 2017Natural TTS Synthesis by Conditioning WaveNet on Mel Spectrogram Predictionscited by 804
  8. 2017VoxCeleb: a large-scale speaker identification datasetcited by 732
  9. 2016Language Modeling with Gated Convolutional Networkscited by 702
  10. 2019Unified Language Model Pre-training for Natural Language Understanding and Generationcited by 670
  11. 2021SUPERB: Speech processing Universal PERformance Benchmarkcited by 666
  12. 2019LibriTTS: A Corpus Derived from LibriSpeech for Text-to-Speechcited by 649
  13. 2018Subword Regularization: Improving Neural Network Translation Models with Multiple Subword Candidatescited by 639
  14. 2018ESPnet: End-to-End Speech Processing Toolkitcited by 558
  15. 2021W2v-BERT: Combining Contrastive Learning and Masked Language Modeling for Self-Supervised Speech Pre-Trainingcited by 347
  16. 2019Parallel WaveGAN: A fast waveform generation model based on generative adversarial networks with multi-resolution spectrogramcited by 340
  17. 2021Generative Spoken Language Modeling from Raw Audiocited by 340
  18. 2020fairseq S2T: Fast Speech-to-Text Modeling with fairseqcited by 176
  19. 2019WHAM!: Extending Speech Separation to Noisy Environmentscited by 156
  20. 2015On Using Monolingual Corpora in Neural Machine Translationcited by 150
  21. 2021Unsupervised Speech Recognitioncited by 148
  22. 2019End-to-end ASR: from Supervised to Semi-Supervised Learning with Modern Architecturescited by 120
  23. 2021Text-Free Prosody-Aware Generative Spoken Language Modelingcited by 119
  24. 2019Generative Pre-Training for Speech with Autoregressive Predictive Codingcited by 82
  25. 2020ESPnet-ST: All-in-One Speech Translation Toolkitcited by 82
  26. 2021UniSpeech: Unified Speech Representation Learning with Labeled and Unlabeled Datacited by 77
  27. 2019Effectiveness of self-supervised pre-training for speech recognitioncited by 72
  28. 2017Efficiently Trainable Text-to-Speech System Based on Deep Convolutional Networks with Guided Attentioncited by 57
  29. 2021Lightweight Adapter Tuning for Multilingual Speech Translationcited by 54
  30. 2021Improving Speech Translation by Understanding and Learning from the Auxiliary Text Translation Taskcited by 54
  31. 2018Speech2Vec: A Sequence-to-Sequence Framework for Learning Word Embeddings from Speechcited by 52
  32. 2020A General Multi-Task Learning Framework to Leverage Text Data for Speech to Text Taskscited by 51
  33. 2021Learning Shared Semantic Space for Speech-to-Text Translationcited by 49
  34. 2021Fused Acoustic and Text Encoding for Multimodal Bilingual Pretraining and Speech Translationcited by 47
  35. 2021End-to-end Speech Translation via Cross-modal Progressive Trainingcited by 47
  36. 2021Deep Learning Based Assessment of Synthetic Speech Naturalnesscited by 35
  37. 2020ICASSP 2021 Deep Noise Suppression Challengecited by 27
  38. 2021Speech-language Pre-training for End-to-end Spoken Language Understandingcited by 27

50 most cited of 59 All 59 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2021: 1; 2022: 27; 2023: 49; 2024: 38; 2025: 43; 2026: 22.

180 papers.

Sort:

By area:

  1. 2021WavLM: Large-Scale Self-Supervised Pre-Training for Full Stack Speech Processingcited by 1,657
  2. 2023MM-Vet: Evaluating Large Multimodal Models for Integrated Capabilitiescited by 1,239
  3. 2023Neural Codec Language Models are Zero-Shot Text to Speech Synthesizerscited by 752
  4. 2022data2vec: A General Framework for Self-supervised Learning in Speech, Vision and Languagecited by 727
  5. 2023Qwen-Audio: Advancing Universal Audio Understanding via Unified Large-Scale Audio-Language Modelscited by 608
  6. 2024Qwen2-Audio Technical Reportcited by 560
  7. 2022AudioGen: Textually Guided Audio Generationcited by 295
  8. 2023End-to-End Speech Recognition: A Surveycited by 129
  9. 2025Advances and Challenges in Foundation Agents: From Brain-Inspired Intelligence to Evolutionary, Collaborative, and Safe Systemscited by 126
  10. 2023ONE-PEACE: Exploring One General Representation Model Toward Unlimited Modalitiescited by 118
  11. 2023Textually Pretrained Speech Language Modelscited by 101
  12. 2023Seeing What You Said: Talking Face Generation Guided by a Lip Reading Expertcited by 98
38 more
  1. 2023Spoken Question Answering and Speech Continuation Using Spectrogram-Powered LLMcited by 94
  2. 2023VioLA: Unified Codec Language Models for Speech Recognition, Synthesis, and Translationcited by 87
  3. 2022MAESTRO: Matched Speech Text Representations through Modality Matchingcited by 79
  4. 2023LauraGPT: Listen, Attend, Understand, and Regenerate Audio with GPTcited by 78
  5. 2023Voxtlm: unified decoder-only models for consolidating speech recognition/synthesis and speech/text continuation taskscited by 72
  6. 2023ChatVideo: A Tracklet-centric Multimodal and Versatile Video Understanding Systemcited by 65
  7. 2023V2A-Mapper: A Lightweight Solution for Vision-to-Audio Generation by Connecting Foundation Modelscited by 65
  8. 2024Autoregressive Speech Synthesis without Vector Quantizationcited by 65
  9. 2022Unified Speech-Text Pre-training for Speech Translation and Recognitioncited by 64
  10. 2024SpeechVerse: A Large-scale Generalizable Audio Language Modelcited by 63
  11. 2023Generative Pre-training for Speech with Flow Matchingcited by 55
  12. 2022LightHuBERT: Lightweight and Configurable Speech Representation Learning with Once-for-All Hidden-Unit BERTcited by 50
  13. 2022SpeechLM: Enhanced Speech Pre-Training with Unpaired Textual Datacited by 49
  14. 2022Enhanced Direct Speech-to-Speech Translation Using Self-supervised Pre-training and Data Augmentationcited by 44
  15. 2022SpeechUT: Bridging Speech and Text with Hidden-Unit for Encoder-Decoder Based Speech-Text Pre-trainingcited by 40
  16. 2023Transformers in Speech Processing: A Surveycited by 39
  17. 2024Distilling an End-to-End Voice Assistant Without Instruction Training Datacited by 39
  18. 2022VATLM: Visual-Audio-Text Pre-Training with Unified Masked Prediction for Speech Representation Learningcited by 37
  19. 2024Foundation Models for Music: A Surveycited by 37
  20. 2024DiTTo-TTS: Diffusion Transformers for Scalable Text-to-Speech without Domain-Specific Factorscited by 32
  21. 2022textless-lib: a Library for Textless Spoken Language Processingcited by 28
  22. 2023Toward Joint Language Modeling for Speech Units and Textcited by 28
  23. 2022Wav2Seq: Pre-training Speech-to-Text Encoder-Decoder Models Using Pseudo Languagescited by 26
  24. 2022Meta Learning for Natural Language Processing: A Surveycited by 26
  25. 2023Pre-training for Speech Translation: CTC Meets Optimal Transportcited by 25
  26. 2022JOIST: A Joint Speech and Text Streaming Model For ASRcited by 24
  27. 2023GPT-FL: Generative Pre-trained Model-Assisted Federated Learningcited by 24
  28. 2023VoiceLDM: Text-to-Speech with Environmental Contextcited by 23
  29. 2024Beyond Language Models: Byte Models are Digital World Simulatorscited by 22
  30. 2024Improving Robustness of LLM-based Speech Synthesis by Learning Monotonic Alignmentcited by 22
  31. 2022WACO: Word-Aligned Contrastive Learning for Speech Translationcited by 21
  32. 2023SpeechGen: Unlocking the Generative Power of Speech Language Models with Promptscited by 21
  33. 2024Speech Robust Bench: A Robustness Benchmark For Speech Recognitioncited by 21
  34. 2025NExT-OMNI: Towards Any-to-Any Omnimodal Foundation Models with Discrete Flow Matchingcited by 21
  35. 2022UAVM: Towards Unifying Audio and Visual Modelscited by 20
  36. 2024HyperLLaVA: Dynamic Visual and Language Expert Tuning for Multimodal Large Language Modelscited by 20
  37. 2023DUB: Discrete Unit Back-translation for Speech Translationcited by 18
  38. 2022OFASys: A Multi-Modal Multi-Task Learning System for Building Generalist Modelscited by 17

50 most cited of 180 All 180 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2021Stacked Acoustic-and-Textual Encoding: Integrating the Pre-trained Models into Speech Translation Encoders
  2. 2025Instituto de Telecomunicacc~oes at IWSLT 2025: Aligning Small-Scale Speech and Language Models for Speech-to-Text Learning
  3. 2026Decoder-only Conformer with Modality-aware Sparse Mixtures of Experts for ASR
  4. 2023Attention or Convolution: Transformer Encoders in Audio Language Models for Inference Efficiency
  5. 2024Speech Recognition Rescoring with Large Speech-Text Foundation Models
  6. 2024VQ-CTAP: Cross-Modal Fine-Grained Sequence Representation Learning for Speech Processing
  7. 2026X-AuT: Progressive Audio-Encoder Compression for Speech LLMs with Cross-Scale Distillation
  8. 2024Discrete Multimodal Transformers with a Pretrained Large Language Model for Mixed-Supervision Speech Processing