mlsanityData to 14 Oct 2026

Wav2Seq: Pre-training Speech-to-Text Encoder-Decoder Models Using Pseudo Languages

arXiv 2205.01086 ↗First posted 2 May 2022cs.CL cs.LG cs.SDcited by 26

Not on the map. Why?

Abstract (arXiv)

We introduce Wav2Seq, the first self-supervised approach to pre-train both parts of encoder-decoder models for speech data. We induce a pseudo language as a compact discrete representation, and formulate a self-supervised pseudo speech recognition task -- transcribing audio inputs into pseudo subword sequences. This process stands on its own, or can be applied as low-cost second-stage pre-training. We experiment with automatic speech recognition (ASR), spoken named entity recognition, and speech-to-text translation. We set new state-of-the-art results for end-to-end spoken named entity recognition, and show consistent improvements on 20 language pairs for speech-to-text translation, even when competing methods use additional text data for training. Finally, on ASR, our approach enables encoder-decoder methods to benefit from pre-training for all parts of the network, and shows comparable performance to highly optimized recent methods.

Key facts. Cited by 26 papers in the map (data through 14 Oct 2026). Builds on 55 of its 74 references that are in the map. First posted on arXiv 2 May 2022.

Builds on

55 of 74 references.

References · 55

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  2. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  3. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,372
  4. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 12,049
  5. 2018Representation Learning with Contrastive Predictive Codingcited by 7,777
  6. 2019Momentum Contrast for Unsupervised Visual Representation Learningcited by 7,513
  7. 2021Emerging Properties in Self-Supervised Vision Transformerscited by 6,968
  8. 2019BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehensioncited by 5,893
  9. 2020Bootstrap your own latent: A new approach to self-supervised Learningcited by 4,750
  10. 2014Neural Machine Translation by Jointly Learning to Align and Translatecited by 4,548
  11. 2020wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representationscited by 3,905
  12. 2014Sequence to Sequence Learning with Neural Networkscited by 2,986
38 more
  1. 2018Deep contextualized word representationscited by 2,290
  2. 2020Conformer: Convolution-augmented Transformer for Speech Recognitioncited by 1,863
  3. 2018Unsupervised Feature Learning via Non-Parametric Instance-level Discriminationcited by 1,845
  4. 2021WavLM: Large-Scale Self-Supervised Pre-Training for Full Stack Speech Processingcited by 1,657
  5. 2019fairseq: A Fast, Extensible Toolkit for Sequence Modelingcited by 1,488
  6. 2021Multitask Prompted Training Enables Zero-Shot Task Generalizationcited by 1,470
  7. 2019PEGASUS: Pre-training with Extracted Gap-sentences for Abstractive Summarizationcited by 1,102
  8. 2020Multilingual Denoising Pre-training for Neural Machine Translationcited by 1,035
  9. 2022data2vec: A General Framework for Self-supervised Learning in Speech, Vision and Languagecited by 727
  10. 2021SUPERB: Speech processing Universal PERformance Benchmarkcited by 666
  11. 2021XLS-R: Self-supervised Cross-lingual Speech Representation Learning at Scalecited by 601
  12. 2018ESPnet: End-to-End Speech Processing Toolkitcited by 558
  13. 2015Listen, Attend and Spellcited by 543
  14. 2020Unsupervised Cross-lingual Representation Learning for Speech Recognitioncited by 501
  15. 2019Libri-Light: A Benchmark for ASR with Limited or No Supervisioncited by 467
  16. 2021VoxPopuli: A Large-Scale Multilingual Speech Corpus for Representation Learning, Semi-Supervised Learning and Interpretationcited by 463
  17. 2015Deep Speech 2: End-to-End Speech Recognition in English and Mandarincited by 451
  18. 2015Attention-Based Models for Speech Recognitioncited by 401
  19. 2021W2v-BERT: Combining Contrastive Learning and Masked Language Modeling for Self-Supervised Speech Pre-Trainingcited by 347
  20. 2019vq-wav2vec: Self-Supervised Learning of Discrete Speech Representationscited by 340
  21. 2021Generative Spoken Language Modeling from Raw Audiocited by 340
  22. 2016Using the Output Embedding to Improve Language Modelscited by 272
  23. 2020ProphetNet: Predicting Future N-gram for Sequence-to-Sequence Pre-trainingcited by 222
  24. 2021SpeechT5: Unified-Modal Encoder-Decoder Pre-Training for Spoken Language Processingcited by 180
  25. 2020fairseq S2T: Fast Speech-to-Text Modeling with fairseqcited by 176
  26. 2021Robust wav2vec 2.0: Analyzing Domain Shift in Self-Supervised Pre-Trainingcited by 151
  27. 2021Efficient Self-supervised Vision Transformers for Representation Learningcited by 149
  28. 2020Self-training and Pre-training are Complementary for Speech Recognitioncited by 99
  29. 2020ContextNet: Improving Convolutional Neural Networks for Automatic Speech Recognition with Global Contextcited by 97
  30. 2016Tying Word Vectors and Word Classifiers: A Loss Framework for Language Modelingcited by 92
  31. 2020ESPnet-ST: All-in-One Speech Translation Toolkitcited by 82
  32. 2015EESEN: End-to-End Speech Recognition using Deep RNN Models and WFST-based Decodingcited by 73
  33. 2021SLUE: New Benchmark Tasks for Spoken Language Understanding Evaluation on Natural Speechcited by 72
  34. 2021Performance-Efficiency Trade-offs in Unsupervised Pre-training for Speech Recognitioncited by 68
  35. 2021PARP: Prune, Adjust and Re-Prune for Self-Supervised Speech Recognitioncited by 53
  36. 2019Transformers with convolutional context for ASRcited by 44
  37. 2016Wav2Letter: an End-to-End ConvNet-based Speech Recognition Systemcited by 43
  38. 2018wav2letter++: The Fastest Open-source Speech Recognition Systemcited by 42

50 most cited of 55 All 55 (JSON)

Built on by

Cited by year

Citing papers per year: 2022: 4; 2023: 9; 2024: 11; 2025: 2.

26 papers.

Sort:

By area:

  1. 2025On The Landscape of Spoken Language Models: A Comprehensive Surveycited by 118
  2. 2023What Do Self-Supervised Speech Models Know About Words?cited by 64
  3. 2022SpeechUT: Bridging Speech and Text with Hidden-Unit for Encoder-Decoder Based Speech-Text Pre-trainingcited by 40
  4. 2023ChatGPT for Shaping the Future of Dentistry: The Potential of Multi-Modal Large Language Modelcited by 40
  5. 2024When Large Language Models Meet Vector Databases: A Surveycited by 39
  6. 2022SLUE Phase-2: A Benchmark Suite of Diverse Spoken Language Understanding Taskscited by 35
  7. 2023Structured Pruning of Self-Supervised Pre-trained Models for Speech Recognition and Understandingcited by 31
  8. 2023Toward Joint Language Modeling for Speech Units and Textcited by 28
  9. 2024Retrieval Augmented End-to-End Spoken Dialog Modelscited by 20
  10. 2023DUB: Discrete Unit Back-translation for Speech Translationcited by 18
  11. 2023Recent Advances in Direct Speech-to-text Translationcited by 15
  12. 2024SpeechPrompt: Prompting Speech Language Models for Speech Processing Taskscited by 12
14 more
  1. 2024Predicting positive transfer for improved low-resource speech recognition using acoustic pseudo-tokenscited by 11
  2. 2024DiscreteSLU: A Large Language Model with Self-Supervised Discrete Speech Units for Spoken Language Understandingcited by 10
  3. 2022MMSpeech: Multi-modal Multi-task Encoder-Decoder Pre-training for Speech Recognitioncited by 9
  4. 2023Speech-to-Text Adapter and Speech-to-Entity Retriever Augmented LLMs for Speech Understandingcited by 9
  5. 2024On the Evaluation of Speech Foundation Models for Spoken Language Understandingcited by 9
  6. 2023R-Spin: Efficient Speaker and Noise-invariant Representation Learning with Acoustic Piecescited by 7
  7. 2023Prompting and Adapter Tuning for Self-supervised Encoder-Decoder Speech Modelcited by 3
  8. 2022Bootstrapping meaning through listening: Unsupervised learning of spoken sentence embeddingscited by 2
  9. 2024UniEnc-CASSNAT: An Encoder-only Non-autoregressive ASR for Speech SSL Modelscited by 1
  10. 2024Compact Speech Translation Models via Discrete Speech Units Pretrainingcited by 1
  11. 2025Leveraging Unit Language Guidance to Advance Speech Modeling in Textless Speech-to-Speech Translationcited by 1
  12. 2024DEPTH: Discourse Education through Pre-Training Hierarchicallycited by 0
  13. 2024Trading Devil Final: Backdoor attack via Stock market and Bayesian Optimizationcited by 0
  14. 2024Trading Devil RL: Backdoor attack via Stock market, Bayesian Optimization and Reinforcement Learningcited by 0