mlsanityData to 14 Oct 2026

Improving End-to-End Speech Translation by Imitation-Based Knowledge Distillation with Synthetic Transcripts

arXiv 2307.08426 ↗First posted 17 Jul 2023cs.CLcited by 1

On the map: Knowledge distillation → Speech and audio distillation

Abstract (arXiv)

End-to-end automatic speech translation (AST) relies on data that combines audio inputs with text translation outputs. Previous work used existing large parallel corpora of transcriptions and translations in a knowledge distillation (KD) setup to distill a neural machine translation (NMT) into an AST student model. While KD allows using larger pretrained models, the reliance of previous KD approaches on manual audio transcripts in the data pipeline restricts the applicability of this framework to AST. We present an imitation learning approach where a teacher NMT system corrects the errors of an AST student without relying on manual transcripts. We show that the NMT teacher can recover from errors in automatic transcriptions and is able to correct erroneous translations of the AST student, leading to improvements of about 4 BLEU points over the standard AST end-to-end baseline on the English-German CoVoST-2 and MuST-C datasets, respectively. Code and data are publicly available.footnoteurlhttps://github.com/HubReb/imitkdast/releases/tag/v1.1

Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Placed in Speech and audio distillation, one of 12 directions in Knowledge distillation. Builds on 35 of its 48 references that are in the map. First posted on arXiv 17 Jul 2023.

Builds on

35 of 48 references.

References · 35

  1. 2017Attention Is All You Needcited by 60,361
  2. 2014Adam: A Method for Stochastic Optimizationcited by 37,240
  3. 2015Distilling the Knowledge in a Neural Networkcited by 10,324
  4. 2015Rethinking the Inception Architecture for Computer Visioncited by 5,475
  5. 2022Robust Speech Recognition via Large-Scale Weak Supervisioncited by 4,057
  6. 2015Neural Machine Translation of Rare Words with Subword Unitscited by 2,918
  7. 2018A Call for Clarity in Reporting BLEU Scorescited by 1,799
  8. 2019fairseq: A Fast, Extensible Toolkit for Sequence Modelingcited by 1,488
  9. 2015Scheduled Sampling for Sequence Prediction with Recurrent Neural Networkscited by 803
  10. 2016Sequence-Level Knowledge Distillationcited by 801
  11. 2015Sequence Level Training with Recurrent Neural Networkscited by 576
  12. 2019Facebook FAIR's WMT19 News Translation Task Submissioncited by 179
23 more
  1. 2020fairseq S2T: Fast Speech-to-Text Modeling with fairseqcited by 176
  2. 2016An Actor-Critic Algorithm for Sequence Predictioncited by 159
  3. 2020On Exposure Bias, Hallucination and Domain Shift in Neural Machine Translationcited by 127
  4. 2019Understanding Knowledge Distillation in Non-autoregressive Machine Translationcited by 122
  5. 2017Sequence-to-Sequence Models Can Directly Translate Foreign Speechcited by 121
  6. 2014Reinforcement and Imitation Learning via Interactive No-Regret Learningcited by 118
  7. 2019Europarl-ST: A Multilingual Corpus For Speech Translation Of Parliamentary Debatescited by 114
  8. 2021Scaling Laws for Neural Machine Translationcited by 111
  9. 2021BigSSL: Exploring the Frontier of Large-Scale Semi-Supervised Learning for Automatic Speech Recognitioncited by 88
  10. 2020Autoregressive Knowledge Distillation through Imitation Learningcited by 74
  11. 2018End-to-End Automatic Speech Translation of Audiobookscited by 65
  12. 2019End-to-End Speech Translation with Knowledge Distillationcited by 58
  13. 2018Leveraging Weakly Supervised Data to Improve End-to-End Speech-to-Text Translationcited by 57
  14. 2018Tied Multitask Learning for Neural Speech Translationcited by 54
  15. 2021Improving Speech Translation by Understanding and Learning from the Auxiliary Text Translation Taskcited by 54
  16. 2020A General Multi-Task Learning Framework to Leverage Text Data for Speech to Text Taskscited by 51
  17. 2021Fused Acoustic and Text Encoding for Multimodal Bilingual Pretraining and Speech Translationcited by 47
  18. 2021End-to-end Speech Translation via Cross-modal Progressive Trainingcited by 47
  19. 2019A Comparative Study on End-to-end Speech to Text Translationcited by 40
  20. 2020Self-Training for End-to-End Speech Translationcited by 36
  21. 2020End-to-End Speech-Translation with Knowledge Distillation: FBK@IWSLT2020cited by 31
  22. 2022Revisiting End-to-End Speech-to-Text Translation From Scratchcited by 26
  23. 2021Fixing exposure bias with imitation learning needs powerful oraclescited by 3

Built on by

Cited by year

Citing papers per year: 2024: 1.

1 paper.

  1. 2024Imitating Language via Scalable Inverse Reinforcement Learningcited by 21

Similar, not cited

Close in wording, no citation either way

  1. 2022Sample, Translate, Recombine: Leveraging Audio Alignments for Data Augmentation in End-to-end Speech Translation
  2. 2021Source and Target Bidirectional Knowledge Distillation for End-to-end Speech Translation
  3. 2023Decouple Non-parametric Knowledge Distillation For End-to-end Speech Translation
  4. 2023Improved Cross-Lingual Transfer Learning For Automatic Speech Translation
  5. 2021Selective Knowledge Distillation for Neural Machine Translation
  6. 2021Stacked Acoustic-and-Textual Encoding: Integrating the Pre-trained Models into Speech Translation Encoders
  7. 2024Leave No Knowledge Behind During Knowledge Distillation: Towards Practical and Effective Knowledge Distillation for Code-Switching ASR Using Realistic Data
  8. 2026Evolving Knowledge Distillation for Lightweight Neural Machine Translation