# Curriculum Pre-training for End-to-End Speech Translation

arXiv: https://arxiv.org/abs/2004.10093 · HTML: https://mlsanity.com/paper/2004.10093/

## Key facts

- Cited by 57 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 21 Apr 2020.

## Built on by (most cited first, 30 of 57)

- [Curriculum Learning: A Survey](https://mlsanity.com/paper/2101.10382/) (2021)
- [Cross-modal Contrastive Learning for Speech Translation](https://mlsanity.com/paper/2205.02444/) (2022)
- [STEMM: Self-learning with Speech-text Manifold Mixup for Speech Translation](https://mlsanity.com/paper/2203.10426/) (2022)
- [Learning Shared Semantic Space for Speech-to-Text Translation](https://mlsanity.com/paper/2105.03095/) (2021)
- [Stacked Acoustic-and-Textual Encoding: Integrating the Pre-trained Models into Speech Translation Encoders](https://mlsanity.com/paper/2105.05752/) (2021)
- [End-to-end Speech Translation via Cross-modal Progressive Training](https://mlsanity.com/paper/2104.10380/) (2021)
- [RealTranS: End-to-End Simultaneous Speech Translation with Convolutional Weighted-Shrinking Transformer](https://mlsanity.com/paper/2106.04833/) (2021)
- [Learning When to Translate for Streaming Speech](https://mlsanity.com/paper/2109.07368/) (2021)
- [Source and Target Bidirectional Knowledge Distillation for End-to-end Speech Translation](https://mlsanity.com/paper/2104.06457/) (2021)
- [Denoising Task Difficulty-based Curriculum for Training Diffusion Models](https://mlsanity.com/paper/2403.10348/) (2024)
- [Pre-training for Speech Translation: CTC Meets Optimal Transport](https://mlsanity.com/paper/2301.11716/) (2023)
- [CLNode: Curriculum Learning for Node Classification](https://mlsanity.com/paper/2206.07258/) (2022)
- [CMOT: Cross-modal Mixup via Optimal Transport for Speech Translation](https://mlsanity.com/paper/2305.14635/) (2023)
- [WACO: Word-Aligned Contrastive Learning for Speech Translation](https://mlsanity.com/paper/2212.09359/) (2022)
- [Regularizing End-to-End Speech Translation with Triangular Decomposition Agreement](https://mlsanity.com/paper/2112.10991/) (2021)
- [Weakly-supervised Temporal Path Representation Learning with Contrastive Curriculum Learning -- Extended Version](https://mlsanity.com/paper/2203.16110/) (2022)
- [Understanding and Bridging the Modality Gap for Speech Translation](https://mlsanity.com/paper/2305.08706/) (2023)
- [Recent Advances in Direct Speech-to-text Translation](https://mlsanity.com/paper/2306.11646/) (2023)
- [Non-Parametric Domain Adaptation for End-to-End Speech Translation](https://mlsanity.com/paper/2205.11211/) (2022)
- [LLaST: Improved End-to-end Speech Translation System Leveraged by Large Language Models](https://mlsanity.com/paper/2407.15415/) (2024)
- [Non-autoregressive End-to-end Speech Translation with Parallel Autoregressive Rescoring](https://mlsanity.com/paper/2109.04411/) (2021)
- [Fast-MD: Fast Multi-Decoder End-to-End Speech Translation with Non-Autoregressive Hidden Intermediates](https://mlsanity.com/paper/2109.12804/) (2021)
- [Improving End-to-end Speech Translation by Leveraging Auxiliary Speech and Text Data](https://mlsanity.com/paper/2212.01778/) (2022)
- [Back Translation for Speech-to-text Translation Without Transcripts](https://mlsanity.com/paper/2305.08709/) (2023)
- [ComSL: A Composite Speech-Language Model for End-to-End Speech-to-Text Translation](https://mlsanity.com/paper/2305.14838/) (2023)
- [CTC-based Non-autoregressive Speech Translation](https://mlsanity.com/paper/2305.17358/) (2023)
- [Tuning Large language model for End-to-end Speech Translation](https://mlsanity.com/paper/2310.02050/) (2023)
- [End-to-End Speech-to-Text Translation: A Survey](https://mlsanity.com/paper/2312.01053/) (2023)
- [Improving Speech Translation by Cross-Modal Multi-Grained Contrastive Learning](https://mlsanity.com/paper/2304.10309/) (2023)
- [Transfer Learning from Pre-trained Language Models Improves End-to-End Speech Summarization](https://mlsanity.com/paper/2306.04233/) (2023)

## Abstract (arXiv)

End-to-end speech translation poses a heavy burden on the encoder, because it has to transcribe, understand, and learn cross-lingual semantics simultaneously. To obtain a powerful encoder, traditional methods pre-train it on ASR data to capture speech features. However, we argue that pre-training the encoder only through simple speech recognition is not enough and high-level linguistic knowledge should be considered. Inspired by this, we propose a curriculum pre-training method that includes an elementary course for transcription learning and two advanced courses for understanding the utterance and mapping words in two languages. The difficulty of these courses is gradually increasing. Experiments show that our curriculum pre-training method leads to significant improvements on En-De and En-Fr speech translation benchmarks.