mlsanityData to 7 Oct 2026

Speech Translation with Foundation Models and Optimal Transport: UPC at IWSLT23

arXiv 2306.01327 ↗First posted 2 Jun 2023cs.CL cs.SD eess.AScited by 1

Not on the map. Why?

Abstract (arXiv)

This paper describes the submission of the UPC Machine Translation group to the IWSLT 2023 Offline Speech Translation task. Our Speech Translation systems utilize foundation models for speech (wav2vec 2.0) and text (mBART50). We incorporate a Siamese pretraining step of the speech and text encoders with CTC and Optimal Transport, to adapt the speech representations to the space of the text model, thus maximizing transfer learning from MT. After this pretraining, we fine-tune our system end-to-end on ST, with Cross Entropy and Knowledge Distillation. Apart from the available ST corpora, we create synthetic data with SegAugment to better adapt our models to the custom segmentations of the IWSLT test sets. Our best single model obtains 31.2 BLEU points on MuST-C tst-COMMON, 29.8 points on IWLST.tst2020 and 33.4 points on the newly released IWSLT.ACLdev2023.

Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Builds on 41 of its 66 references that are in the map. First posted on arXiv 2 Jun 2023.

Builds on

41 of 66 references.

References · 41

  1. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  2. 2015Distilling the Knowledge in a Neural Networkcited by 10,076
  3. 2019Parameter-Efficient Transfer Learning for NLPcited by 4,195
  4. 2020wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representationscited by 3,853
  5. 2016Gaussian Error Linear Units (GELUs)cited by 3,431
  6. 2021HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Unitscited by 2,301
  7. 2018A Call for Clarity in Reporting BLEU Scorescited by 1,790
  8. 2019fairseq: A Fast, Extensible Toolkit for Sequence Modelingcited by 1,486
  9. 2019Common Voice: A Massively-Multilingual Speech Corpuscited by 1,358
  10. 2020COMET: A Neural Framework for MT Evaluationcited by 1,024
  11. 2020On Layer Normalization in the Transformer Architecturecited by 906
  12. 2021Towards a Unified View of Parameter-Efficient Transfer Learningcited by 863
29 more
  1. 2020Unsupervised Cross-lingual Representation Learning for Speech Recognitioncited by 496
  2. 2019Libri-Light: A Benchmark for ASR with Limited or No Supervisioncited by 462
  3. 2020Multilingual Translation with Extensible Multilingual Pretraining and Finetuningcited by 287
  4. 2015Learning with a Wasserstein Losscited by 194
  5. 2020fairseq S2T: Fast Speech-to-Text Modeling with fairseqcited by 174
  6. 2021Robust wav2vec 2.0: Analyzing Domain Shift in Self-Supervised Pre-Trainingcited by 148
  7. 2019Europarl-ST: A Multilingual Corpus For Speech Translation Of Parliamentary Debatescited by 114
  8. 2020Self-training and Pre-training are Complementary for Speech Recognitioncited by 98
  9. 2019Multilingual Neural Machine Translation with Knowledge Distillationcited by 95
  10. 2020Speech Translation and the End-to-End Promise: Taking Stock of Where We Arecited by 77
  11. 2020CoVoST 2 and Massively Multilingual Speech-to-Text Translationcited by 73
  12. 2018Pre-training on high-resource speech recognition improves low-resource speech-to-text translationcited by 70
  13. 2018End-to-End Automatic Speech Translation of Audiobookscited by 65
  14. 2020Data Augmenting Contrastive Learning of Speech Representations in the Time Domaincited by 63
  15. 2019End-to-End Speech Translation with Knowledge Distillationcited by 58
  16. 2021Cascade versus Direct Speech Translation: Do the Differences Still Make a Difference?cited by 54
  17. 2021Learning Shared Semantic Space for Speech-to-Text Translationcited by 49
  18. 2022SpeechLM: Enhanced Speech Pre-Training with Unpaired Textual Datacited by 49
  19. 2021Stacked Acoustic-and-Textual Encoding: Integrating the Pre-trained Models into Speech Translation Encoderscited by 48
  20. 2021CTC-based Compression for Direct Speech Translationcited by 45
  21. 2020End-to-End Speech-Translation with Knowledge Distillation: FBK@IWSLT2020cited by 31
  22. 2022SHAS: Approaching optimal Segmentation for End-to-End Speech Translationcited by 26
  23. 2020Adaptive Feature Selection for End-to-End Speech Translationcited by 25
  24. 2023Pre-training for Speech Translation: CTC Meets Optimal Transportcited by 25
  25. 2021End-to-End Speech Translation with Pre-trained Models and Adapters: UPC at IWSLT 2021cited by 16
  26. 2022M-Adapter: Modality Adaptation for End-to-End Speech-to-Text Translationcited by 15
  27. 2022Efficient yet Competitive Speech Translation: FBK@IWSLT2022cited by 14
  28. 2022SegAugment: Maximizing the Utility of Speech Translation Data with Segmentation-based Augmentationscited by 6
  29. 2022The YiTrans End-to-End Speech Translation System for IWSLT 2022 Offline Shared Taskcited by 3

Built on by

Cited by year

Citing papers per year: 2024: 1.

1 paper.

  1. 2024Pushing the Limits of Zero-shot End-to-End Speech Translationcited by 14