mlsanityData to 7 Oct 2026

WACO: Word-Aligned Contrastive Learning for Speech Translation

arXiv 2212.09359 ↗First posted 19 Dec 2022cs.CL cs.SD eess.AScited by 21

Not on the map. Why?

Abstract (arXiv)

End-to-end Speech Translation (E2E ST) aims to directly translate source speech into target text. Existing ST methods perform poorly when only extremely small speech-text data are available for training. We observe that an ST model's performance closely correlates with its embedding similarity between speech and source transcript. In this paper, we propose Word-Aligned COntrastive learning (WACO), a simple and effective method for extremely low-resource speech-to-text translation. Our key idea is bridging word-level representations for both speech and text modalities via contrastive learning. We evaluate WACO and other methods on the MuST-C dataset, a widely used ST benchmark, and on a low-resource direction Maltese-English from IWSLT 2023. Our experiments demonstrate that WACO outperforms the best baseline by 9+ BLEU points with only 1-hour parallel ST data. Code is available at https://github.com/owaski/WACO.

Key facts. Cited by 21 papers in the map (data through 7 Oct 2026). Builds on 39 of its 55 references that are in the map. First posted on arXiv 19 Dec 2022.

Builds on

39 of 55 references.

References · 39

  1. 2017Attention Is All You Needcited by 59,932
  2. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  3. 2020wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representationscited by 3,853
  4. 2018SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processingcited by 1,973
  5. 2018A Call for Clarity in Reporting BLEU Scorescited by 1,790
  6. 2019Common Voice: A Massively-Multilingual Speech Corpuscited by 1,358
  7. 2019SpecAugment: A Simple Data Augmentation Method for Automatic Speech Recognitioncited by 1,355
  8. 2022No Language Left Behind: Scaling Human-Centered Machine Translationcited by 1,290
  9. 2016Sequence-Level Knowledge Distillationcited by 761
  10. 2021SpeechT5: Unified-Modal Encoder-Decoder Pre-Training for Spoken Language Processingcited by 177
  11. 2017Sequence-to-Sequence Models Can Directly Translate Foreign Speechcited by 121
  12. 2016Listen and Translate: A Proof of Concept for End-to-End Speech-to-Text Translationcited by 113
27 more
  1. 2018Pre-training on high-resource speech recognition improves low-resource speech-to-text translationcited by 70
  2. 2021SLAM: A Unified Encoder for Speech and Language Modeling via Speech-Text Joint Pre-Trainingcited by 69
  3. 2018End-to-End Automatic Speech Translation of Audiobookscited by 65
  4. 2022Unified Speech-Text Pre-training for Speech Translation and Recognitioncited by 64
  5. 2022Cross-modal Contrastive Learning for Speech Translationcited by 61
  6. 2018Leveraging Weakly Supervised Data to Improve End-to-End Speech-to-Text Translationcited by 57
  7. 2020Curriculum Pre-training for End-to-End Speech Translationcited by 57
  8. 2022STEMM: Self-learning with Speech-text Manifold Mixup for Speech Translationcited by 57
  9. 2020Bridging the Modality Gap for Speech-to-Text Translationcited by 55
  10. 2020Dual-decoder Transformer for Joint Automatic Speech Recognition and Multilingual Speech Translationcited by 54
  11. 2021Improving Speech Translation by Understanding and Learning from the Auxiliary Text Translation Taskcited by 54
  12. 2020A General Multi-Task Learning Framework to Leverage Text Data for Speech to Text Taskscited by 51
  13. 2021Learning Shared Semantic Space for Speech-to-Text Translationcited by 49
  14. 2021Stacked Acoustic-and-Textual Encoding: Integrating the Pre-trained Models into Speech Translation Encoderscited by 48
  15. 2021Fused Acoustic and Text Encoding for Multimodal Bilingual Pretraining and Speech Translationcited by 47
  16. 2021End-to-end Speech Translation via Cross-modal Progressive Trainingcited by 47
  17. 2022SpeechUT: Bridging Speech and Text with Hidden-Unit for Encoder-Decoder Based Speech-Text Pre-trainingcited by 40
  18. 2020Self-Training for End-to-End Speech Translationcited by 36
  19. 2020End-to-End Speech-Translation with Knowledge Distillation: FBK@IWSLT2020cited by 31
  20. 2020Consecutive Decoding for Speech-to-text Translationcited by 28
  21. 2022Sample, Translate, Recombine: Leveraging Audio Alignments for Data Augmentation in End-to-end Speech Translationcited by 27
  22. 2022Revisiting End-to-End Speech-to-Text Translation From Scratchcited by 26
  23. 2020Self-Supervised Representations Improve End-to-End Speech Translationcited by 21
  24. 2021End-to-End Spoken Language Understanding for Generalized Voice Assistantscited by 17
  25. 2020SkinAugment: Auto-Encoding Speaker Conversions for Automatic Speech Translationcited by 13
  26. 2022Non-Parametric Domain Adaptation for End-to-End Speech Translationcited by 12
  27. 2022Discrete Cross-Modal Alignment Enables Zero-Shot Speech Translationcited by 6

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 7; 2024: 4; 2025: 8; 2026: 2.

21 papers.

Sort:

  1. 2024Heterogeneous Contrastive Learning for Foundation Models and Beyondcited by 27
  2. 2023Pre-training for Speech Translation: CTC Meets Optimal Transportcited by 25
  3. 2024Pushing the Limits of Zero-shot End-to-End Speech Translationcited by 14
  4. 2023ComSL: A Composite Speech-Language Model for End-to-End Speech-to-Text Translationcited by 9
  5. 2023Tuning Large language model for End-to-end Speech Translationcited by 9
  6. 2024Contrastive Learning for Task-Independent SpeechLLM-Pretrainingcited by 8
  7. 2024FASST: Fast LLM-based Simultaneous Speech Translationcited by 7
  8. 2023Modality Adaption or Regularization? A Case Study on End-to-End Speech Translationcited by 6
  9. 2023An Empirical Study of Consistency Regularization for End-to-End Speech-to-Text Translationcited by 6
  10. 2023Soft Alignment of Modality Space for End-to-end Speech Translationcited by 4
  11. 2023Cross-Modal Multi-Tasking for Speech-to-Text Translation via Hard Parameter Sharingcited by 3
  12. 2025Speech Translation Refinement using Large Language Modelscited by 2
9 more
  1. 2025Adaptive Inner Speech-Text Alignment for LLM-based Speech Translationcited by 2
  2. 2025Optimal Transport Regularization for Speech Text Alignment in Spoken Language Modelscited by 2
  3. 2025DTW-Align: Bridging the Modality Gap in End-to-End Speech Translation with Dynamic Time Warping Alignmentcited by 2
  4. 2025Improving Language and Modality Transfer in Translation by Character-level Modelingcited by 1
  5. 2025Interpretable Style Takagi-Sugeno-Kang Fuzzy Clusteringcited by 0
  6. 2025Optimal Multi-Task Learning at Regularization Horizon for Speech Translation Taskcited by 0
  7. 2025POTSA: A Cross-Lingual Speech Alignment Framework for Speech-to-Text Translationcited by 0
  8. 2026Cross-Modal Robustness Transfer (CMRT): Training Robust Speech Translation Models Using Adversarial Textcited by 0
  9. 2026Align, Integrate, and Fire: Efficient Token-Level Alignment for Zero-Shot SpeechLLMscited by 0