mlsanityData to 7 Oct 2026

End-to-End Speech Translation with Pre-trained Models and Adapters: UPC at IWSLT 2021

arXiv 2105.04512 ↗First posted 10 May 2021cs.CLcited by 16

Not on the map. Why?

Abstract (arXiv)

This paper describes the submission to the IWSLT 2021 offline speech translation task by the UPC Machine Translation group. The task consists of building a system capable of translating English audio recordings extracted from TED talks into German text. Submitted systems can be either cascade or end-to-end and use a custom or given segmentation. Our submission is an end-to-end speech translation system, which combines pre-trained models (Wav2Vec 2.0 and mBART) with coupling modules between the encoder and decoder, and uses an efficient fine-tuning technique, which trains only 20% of its total parameters. We show that adding an Adapter to the system and pre-training it, can increase the convergence speed and the final result, with which we achieve a BLEU score of 27.3 on the MuST-C test set. Our final model is an ensemble that obtains 28.22 BLEU score on the same set. Our submission also uses a custom segmentation algorithm that employs pre-trained Wav2Vec 2.0 for identifying periods of untranscribable text and can bring improvements of 2.5 to 3 BLEU score on the IWSLT 2019 test set, as compared to the result with the given segmentation.

Key facts. Cited by 16 papers in the map (data through 7 Oct 2026). Builds on 23 of its 38 references that are in the map. First posted on arXiv 10 May 2021.

Builds on

23 of 38 references.

References · 23

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  4. 2019BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehensioncited by 5,874
  5. 2016Layer Normalizationcited by 5,579
  6. 2020wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representationscited by 3,853
  7. 2018A Call for Clarity in Reporting BLEU Scorescited by 1,790
  8. 2019SpecAugment: A Simple Data Augmentation Method for Automatic Speech Recognitioncited by 1,355
  9. 2020Multilingual Denoising Pre-training for Neural Machine Translationcited by 1,028
  10. 2019Libri-Light: A Benchmark for ASR with Limited or No Supervisioncited by 462
  11. 2020Multilingual Translation with Extensible Multilingual Pretraining and Finetuningcited by 287
  12. 2019Simple, Scalable Adaptation for Neural Machine Translationcited by 239
11 more
  1. 2020fairseq S2T: Fast Speech-to-Text Modeling with fairseqcited by 174
  2. 2017Sequence-to-Sequence Models Can Directly Translate Foreign Speechcited by 121
  3. 2019Europarl-ST: A Multilingual Corpus For Speech Translation Of Parliamentary Debatescited by 114
  4. 2020Self-training and Pre-training are Complementary for Speech Recognitioncited by 98
  5. 2020CoVoST 2 and Massively Multilingual Speech-to-Text Translationcited by 73
  6. 2018Pre-training on high-resource speech recognition improves low-resource speech-to-text translationcited by 70
  7. 2018End-to-End Automatic Speech Translation of Audiobookscited by 65
  8. 2020Data Augmenting Contrastive Learning of Speech Representations in the Time Domaincited by 63
  9. 2020End-to-End Speech-Translation with Knowledge Distillation: FBK@IWSLT2020cited by 31
  10. 2019On Using SpecAugment for End-to-End Speech Translationcited by 19
  11. 2020Enabling Zero-shot Multilingual Spoken Language Translation with Language-Specific Encoders and Decoderscited by 10

Built on by

Cited by year

Citing papers per year: 2021: 1; 2022: 10; 2023: 3; 2024: 2.

16 papers.

Sort:

  1. 2021Lightweight Adapter Tuning for Multilingual Speech Translationcited by 53
  2. 2022SHAS: Approaching optimal Segmentation for End-to-End Speech Translationcited by 26
  3. 2022M-Adapter: Modality Adaptation for End-to-End Speech-to-Text Translationcited by 15
  4. 2024Pushing the Limits of Zero-shot End-to-End Speech Translationcited by 14
  5. 2022Towards Relation Extraction From Speechcited by 10
  6. 2022Improving End-to-end Speech Translation by Leveraging Auxiliary Speech and Text Datacited by 9
  7. 2023End-to-End Speech-to-Text Translation: A Surveycited by 9
  8. 2022Speech Segmentation Optimization using Segmented Bilingual Speech Corpus for End-to-end Speech Translationcited by 7
  9. 2022SegAugment: Maximizing the Utility of Speech Translation Data with Segmentation-based Augmentationscited by 6
  10. 2022Generating Synthetic Speech from SpokenVocab for Speech Translationcited by 5
  11. 2023Long-form Simultaneous Speech Translation: Thesis Proposalcited by 3
  12. 2022A High-Quality and Large-Scale Dataset for English-Vietnamese Speech Translationcited by 2
4 more
  1. 2022RedApt: An Adaptor for wav2vec 2 Encoding Faster and Smaller Speech Translation without Quality Compromisecited by 2
  2. 2022Self-supervised Rewiring of Pre-trained Speech Encoders: Towards Faster Fine-tuning with Less Labels in Speech Processingcited by 2
  3. 2023Speech Translation with Foundation Models and Optimal Transport: UPC at IWSLT23cited by 1
  4. 2024Lightweight Audio Segmentation for Long-form Speech Translationcited by 1