# Sample, Translate, Recombine: Leveraging Audio Alignments for Data Augmentation in End-to-end Speech Translation

arXiv: https://arxiv.org/abs/2203.08757 · HTML: https://mlsanity.com/paper/2203.08757/

## Key facts

- Cited by 27 papers in the map (data through 14 Oct 2026).
- Placed in Speech and audio distillation, one of 12 directions in Knowledge distillation.
- Builds on 16 of its 28 references that are in the map.
- First posted on arXiv 16 Mar 2022.

## Placed in

- Knowledge distillation / [Speech and audio distillation](https://mlsanity.com/distill/speech-and-audio-distillation/)

## Builds on (most cited first, 16 of 16)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [CutMix: Regularization Strategy to Train Strong Classifiers with Localizable Features](https://mlsanity.com/paper/1905.04899/) (2019)
- [SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processing](https://mlsanity.com/paper/1808.06226/) (2018)
- [A Call for Clarity in Reporting BLEU Scores](https://mlsanity.com/paper/1804.08771/) (2018)
- [fairseq: A Fast, Extensible Toolkit for Sequence Modeling](https://mlsanity.com/paper/1904.01038/) (2019)
- [SpecAugment: A Simple Data Augmentation Method for Automatic Speech Recognition](https://mlsanity.com/paper/1904.08779/) (2019)
- [WikiMatrix: Mining 135M Parallel Sentences in 1620 Language Pairs from Wikipedia](https://mlsanity.com/paper/1907.05791/) (2019)
- [fairseq S2T: Fast Speech-to-Text Modeling with fairseq](https://mlsanity.com/paper/2010.05171/) (2020)
- [Europarl-ST: A Multilingual Corpus For Speech Translation Of Parliamentary Debates](https://mlsanity.com/paper/1911.03167/) (2019)
- [Speech Translation and the End-to-End Promise: Taking Stock of Where We Are](https://mlsanity.com/paper/2004.06358/) (2020)
- [Leveraging Weakly Supervised Data to Improve End-to-End Speech-to-Text Translation](https://mlsanity.com/paper/1811.02050/) (2018)
- [Source and Target Bidirectional Knowledge Distillation for End-to-end Speech Translation](https://mlsanity.com/paper/2104.06457/) (2021)
- [SkinAugment: Auto-Encoding Speaker Conversions for Automatic Speech Translation](https://mlsanity.com/paper/2002.12231/) (2020)
- [Cascaded Models With Cyclic Feedback For Direct Speech Translation](https://mlsanity.com/paper/2010.11153/) (2020)
- [On-the-Fly Aligned Data Augmentation for Sequence-to-Sequence ASR](https://mlsanity.com/paper/2104.01393/) (2021)

## Built on by (most cited first, 27 of 27)

- [CMOT: Cross-modal Mixup via Optimal Transport for Speech Translation](https://mlsanity.com/paper/2305.14635/) (2023)
- [M3ST: Mix at Three Levels for Speech Translation](https://mlsanity.com/paper/2212.03657/) (2022)
- [WACO: Word-Aligned Contrastive Learning for Speech Translation](https://mlsanity.com/paper/2212.09359/) (2022)
- [StreamAtt: Direct Streaming Speech-to-Text Translation with Attention-based Audio History Selection](https://mlsanity.com/paper/2406.06097/) (2024)
- [Understanding and Bridging the Modality Gap for Speech Translation](https://mlsanity.com/paper/2305.08706/) (2023)
- [Recent Advances in Direct Speech-to-text Translation](https://mlsanity.com/paper/2306.11646/) (2023)
- [Pushing the Limits of Zero-shot End-to-End Speech Translation](https://mlsanity.com/paper/2402.10422/) (2024)
- [LLaST: Improved End-to-end Speech Translation System Leveraged by Large Language Models](https://mlsanity.com/paper/2407.15415/) (2024)
- [Back Translation for Speech-to-text Translation Without Transcripts](https://mlsanity.com/paper/2305.08709/) (2023)
- [ComSL: A Composite Speech-Language Model for End-to-End Speech-to-Text Translation](https://mlsanity.com/paper/2305.14838/) (2023)
- [Tuning Large language model for End-to-end Speech Translation](https://mlsanity.com/paper/2310.02050/) (2023)
- [Aspect-Based Sentiment Analysis with Explicit Sentiment Augmentations](https://mlsanity.com/paper/2312.10961/) (2023)
- [Make More of Your Data: Minimal Effort Data Augmentation for Automatic Speech Recognition and Translation](https://mlsanity.com/paper/2210.15398/) (2022)
- [SegAugment: Maximizing the Utility of Speech Translation Data with Segmentation-based Augmentations](https://mlsanity.com/paper/2212.09699/) (2022)
- [An Empirical Study of Consistency Regularization for End-to-End Speech-to-Text Translation](https://mlsanity.com/paper/2308.14482/) (2023)
- [Generating Synthetic Speech from SpokenVocab for Speech Translation](https://mlsanity.com/paper/2210.08174/) (2022)
- [Joint Speech Translation and Named Entity Recognition](https://mlsanity.com/paper/2210.11987/) (2022)
- [Prepending or Cross-Attention for Speech-to-Text? An Empirical Comparison](https://mlsanity.com/paper/2501.02370/) (2025)
- [A Unified Framework for Generative Data Augmentation: A Comprehensive Survey](https://mlsanity.com/paper/2310.00277/) (2023)
- [Ensembling Neural Networks for Improved Prediction and Privacy in Early Diagnosis of Sepsis](https://mlsanity.com/paper/2209.00439/) (2022)
- [Towards a Deep Understanding of Multilingual End-to-End Speech Translation](https://mlsanity.com/paper/2310.20456/) (2023)
- [Representation Purification for End-to-End Speech Translation](https://mlsanity.com/paper/2412.04266/) (2024)
- [Speech Translation Refinement using Large Language Models](https://mlsanity.com/paper/2501.15090/) (2025)
- [Efficient Speech Translation with Pre-trained Models](https://mlsanity.com/paper/2211.04939/) (2022)
- [SpeechT: Findings of the First Mentorship in Speech Translation](https://mlsanity.com/paper/2502.12050/) (2025)
- [UniPSDA: Unsupervised Pseudo Semantic Data Augmentation for Zero-Shot Cross-Lingual Natural Language Understanding](https://mlsanity.com/paper/2406.16372/) (2024)
- [Interpretable Style Takagi-Sugeno-Kang Fuzzy Clustering](https://mlsanity.com/paper/2504.05125/) (2025)

## Abstract (arXiv)

End-to-end speech translation relies on data that pair source-language speech inputs with corresponding translations into a target language. Such data are notoriously scarce, making synthetic data augmentation by back-translation or knowledge distillation a necessary ingredient of end-to-end training. In this paper, we present a novel approach to data augmentation that leverages audio alignments, linguistic properties, and translation. First, we augment a transcription by sampling from a suffix memory that stores text and audio data. Second, we translate the augmented transcript. Finally, we recombine concatenated audio segments and the generated translation. Besides training an MT-system, we only use basic off-the-shelf components without fine-tuning. While having similar resource demands as knowledge distillation, adding our method delivers consistent improvements of up to 0.9 and 1.1 BLEU points on five language pairs on CoVoST 2 and on two language pairs on Europarl-ST, respectively.