mlsanityData to 7 Oct 2026

Source and Target Bidirectional Knowledge Distillation for End-to-end Speech Translation

arXiv 2104.06457 ↗First posted 13 Apr 2021cs.CL cs.SD eess.AScited by 27

Not on the map. Why?

Abstract (arXiv)

A conventional approach to improving the performance of end-to-end speech translation (E2E-ST) models is to leverage the source transcription via pre-training and joint training with automatic speech recognition (ASR) and neural machine translation (NMT) tasks. However, since the input modalities are different, it is difficult to leverage source language text successfully. In this work, we focus on sequence-level knowledge distillation (SeqKD) from external text-based NMT models. To leverage the full potential of the source language information, we propose backward SeqKD, SeqKD from a target-to-source backward NMT model. To this end, we train a bilingual E2E-ST model to predict paraphrased transcriptions as an auxiliary task with a single decoder. The paraphrases are generated from the translations in bitext via back-translation. We further propose bidirectional SeqKD in which SeqKD from both forward and backward NMT models is combined. Experimental evaluations on both autoregressive and non-autoregressive models show that SeqKD in each direction consistently improves the translation performance, and the effectiveness is complementary regardless of the model capacity.

Key facts. Cited by 27 papers in the map (data through 7 Oct 2026). Builds on 40 of its 57 references that are in the map. First posted on arXiv 13 Apr 2021.

Builds on

40 of 57 references.

References · 40

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  4. 2015Rethinking the Inception Architecture for Computer Visioncited by 5,451
  5. 2015Neural Machine Translation of Rare Words with Subword Unitscited by 2,898
  6. 2020Conformer: Convolution-augmented Transformer for Speech Recognitioncited by 1,838
  7. 2019SpecAugment: A Simple Data Augmentation Method for Automatic Speech Recognitioncited by 1,355
  8. 2019Cross-lingual Language Model Pretrainingcited by 1,088
  9. 2016Sequence-Level Knowledge Distillationcited by 761
  10. 2018Subword Regularization: Improving Neural Network Translation Models with Multiple Subword Candidatescited by 632
  11. 2017Non-Autoregressive Neural Machine Translationcited by 457
  12. 2019A Comparative Study on Transformer vs RNN in Speech Applicationscited by 210
28 more
  1. 2020fairseq S2T: Fast Speech-to-Text Modeling with fairseqcited by 174
  2. 2017Sequence-to-Sequence Models Can Directly Translate Foreign Speechcited by 121
  3. 2019Understanding Knowledge Distillation in Non-autoregressive Machine Translationcited by 121
  4. 2020Automatic Machine Translation Evaluation in Many Languages via Zero-Shot Paraphrasingcited by 114
  5. 2016Listen and Translate: A Proof of Concept for End-to-End Speech-to-Text Translationcited by 113
  6. 2020ESPnet-ST: All-in-One Speech Translation Toolkitcited by 82
  7. 2020CoVoST 2 and Massively Multilingual Speech-to-Text Translationcited by 73
  8. 2018Pre-training on high-resource speech recognition improves low-resource speech-to-text translationcited by 70
  9. 2018End-to-End Automatic Speech Translation of Audiobookscited by 65
  10. 2019End-to-End Speech Translation with Knowledge Distillationcited by 58
  11. 2020Curriculum Pre-training for End-to-End Speech Translationcited by 57
  12. 2020Understanding and Improving Lexical Choice in Non-Autoregressive Translationcited by 55
  13. 2018Tied Multitask Learning for Neural Speech Translationcited by 54
  14. 2020Dual-decoder Transformer for Joint Automatic Speech Recognition and Multilingual Speech Translationcited by 54
  15. 2017ParaNMT-50M: Pushing the Limits of Paraphrastic Sentence Embeddings with Millions of Machine Translationscited by 44
  16. 2019A Comparative Study on End-to-end Speech to Text Translationcited by 40
  17. 2019Bridging the Gap between Pre-Training and Fine-Tuning for End-to-End Speech Translationcited by 39
  18. 2019ParaBank: Monolingual Bitext Generation and Sentential Paraphrasing via Lexically-constrained Neural Machine Translationcited by 38
  19. 2019Multilingual End-to-End Speech Translationcited by 37
  20. 2020Semi-Autoregressive Training Improves Mask-Predict Decodingcited by 33
  21. 2020A Study of Non-autoregressive Model for Sequence Generationcited by 31
  22. 2019Attention-Passing Models for Robust and Data-Efficient End-to-End Speech Translationcited by 30
  23. 2017Learning Paraphrastic Sentence Embeddings from Back-Translated Bitextcited by 27
  24. 2020Simulated Multiple Reference Training Improves Low-Resource Machine Translationcited by 22
  25. 2019Synchronous Speech Recognition and Speech-to-Text Translation with Interactive Decodingcited by 21
  26. 2020Orthros: Non-autoregressive End-to-end Speech Translation with Dual-decodercited by 17
  27. 2019Explaining Sequence-Level Knowledge Distillation as Data-Augmentation for Neural Machine Translationcited by 16
  28. 2018Paraphrases as Foreign Languages in Multilingual Neural Machine Translationcited by 5

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 3; 2022: 8; 2023: 11; 2024: 2; 2025: 1; 2026: 2.

27 papers.

Sort:

By area:

  1. 2022Cross-modal Contrastive Learning for Speech Translationcited by 61
  2. 2022UnitY: Two-pass Direct Speech-to-speech Translation with Discrete Unitscited by 60
  3. 2022STEMM: Self-learning with Speech-text Manifold Mixup for Speech Translationcited by 57
  4. 2022Sample, Translate, Recombine: Leveraging Audio Alignments for Data Augmentation in End-to-end Speech Translationcited by 27
  5. 2023Pre-training for Speech Translation: CTC Meets Optimal Transportcited by 25
  6. 2023CMOT: Cross-modal Mixup via Optimal Transport for Speech Translationcited by 22
  7. 2023Understanding and Bridging the Modality Gap for Speech Translationcited by 15
  8. 2023Recent Advances in Direct Speech-to-text Translationcited by 15
  9. 2022Efficient yet Competitive Speech Translation: FBK@IWSLT2022cited by 14
  10. 2023ESPnet-ST-v2: Multipurpose Spoken Language Translation Toolkitcited by 10
  11. 2021Non-autoregressive End-to-end Speech Translation with Parallel Autoregressive Rescoringcited by 9
  12. 2021Fast-MD: Fast Multi-Decoder End-to-End Speech Translation with Non-Autoregressive Hidden Intermediatescited by 9
15 more
  1. 2023ComSL: A Composite Speech-Language Model for End-to-End Speech-to-Text Translationcited by 9
  2. 2023Tuning Large language model for End-to-end Speech Translationcited by 9
  3. 2022Discrete Cross-Modal Alignment Enables Zero-Shot Speech Translationcited by 6
  4. 2022Make More of Your Data: Minimal Effort Data Augmentation for Automatic Speech Recognition and Translationcited by 6
  5. 2023An Empirical Study of Consistency Regularization for End-to-End Speech-to-Text Translationcited by 6
  6. 2022Generating Synthetic Speech from SpokenVocab for Speech Translationcited by 5
  7. 2023Duplex Diffusion Models Improve Speech-to-Speech Translationcited by 5
  8. 2025Prepending or Cross-Attention for Speech-to-Text? An Empirical Comparisoncited by 4
  9. 2023Improving speech translation by fusing speech and textcited by 3
  10. 2023Cross-Modal Multi-Tasking for Speech-to-Text Translation via Hard Parameter Sharingcited by 3
  11. 2024Sentence-wise Speech Summarization: Task, Datasets, and End-to-End Modeling with LM Knowledge Distillationcited by 3
  12. 2024Compact Speech Translation Models via Discrete Speech Units Pretrainingcited by 1
  13. 2021ESPnet-ST IWSLT 2021 Offline Speech Translation Systemcited by 0
  14. 2026Cross-Modal Robustness Transfer (CMRT): Training Robust Speech Translation Models Using Adversarial Textcited by 0
  15. 2026Align, Integrate, and Fire: Efficient Token-Level Alignment for Zero-Shot SpeechLLMscited by 0