mlsanityData to 7 Oct 2026

Sample, Translate, Recombine: Leveraging Audio Alignments for Data Augmentation in End-to-end Speech Translation

arXiv 2203.08757 ↗First posted 16 Mar 2022cs.CL eess.AScited by 27

Not on the map. Why?

Abstract (arXiv)

End-to-end speech translation relies on data that pair source-language speech inputs with corresponding translations into a target language. Such data are notoriously scarce, making synthetic data augmentation by back-translation or knowledge distillation a necessary ingredient of end-to-end training. In this paper, we present a novel approach to data augmentation that leverages audio alignments, linguistic properties, and translation. First, we augment a transcription by sampling from a suffix memory that stores text and audio data. Second, we translate the augmented transcript. Finally, we recombine concatenated audio segments and the generated translation. Besides training an MT-system, we only use basic off-the-shelf components without fine-tuning. While having similar resource demands as knowledge distillation, adding our method delivers consistent improvements of up to 0.9 and 1.1 BLEU points on five language pairs on CoVoST 2 and on two language pairs on Europarl-ST, respectively.

Key facts. Cited by 27 papers in the map (data through 7 Oct 2026). Builds on 16 of its 28 references that are in the map. First posted on arXiv 16 Mar 2022.

Builds on

16 of 28 references.

References · 16

  1. 2017Attention Is All You Needcited by 59,932
  2. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  3. 2019CutMix: Regularization Strategy to Train Strong Classifiers with Localizable Featurescited by 2,875
  4. 2018SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processingcited by 1,973
  5. 2018A Call for Clarity in Reporting BLEU Scorescited by 1,790
  6. 2019fairseq: A Fast, Extensible Toolkit for Sequence Modelingcited by 1,486
  7. 2019SpecAugment: A Simple Data Augmentation Method for Automatic Speech Recognitioncited by 1,355
  8. 2019WikiMatrix: Mining 135M Parallel Sentences in 1620 Language Pairs from Wikipediacited by 221
  9. 2020fairseq S2T: Fast Speech-to-Text Modeling with fairseqcited by 174
  10. 2019Europarl-ST: A Multilingual Corpus For Speech Translation Of Parliamentary Debatescited by 114
  11. 2020Speech Translation and the End-to-End Promise: Taking Stock of Where We Arecited by 77
  12. 2018Leveraging Weakly Supervised Data to Improve End-to-End Speech-to-Text Translationcited by 57
4 more
  1. 2021Source and Target Bidirectional Knowledge Distillation for End-to-end Speech Translationcited by 27
  2. 2020SkinAugment: Auto-Encoding Speaker Conversions for Automatic Speech Translationcited by 13
  3. 2020Cascaded Models With Cyclic Feedback For Direct Speech Translationcited by 9
  4. 2021On-the-Fly Aligned Data Augmentation for Sequence-to-Sequence ASRcited by 9

Built on by

Cited by year

Citing papers per year: 2022: 8; 2023: 10; 2024: 5; 2025: 4.

27 papers.

Sort:

  1. 2023CMOT: Cross-modal Mixup via Optimal Transport for Speech Translationcited by 22
  2. 2022M3ST: Mix at Three Levels for Speech Translationcited by 21
  3. 2022WACO: Word-Aligned Contrastive Learning for Speech Translationcited by 21
  4. 2024StreamAtt: Direct Streaming Speech-to-Text Translation with Attention-based Audio History Selectioncited by 19
  5. 2023Understanding and Bridging the Modality Gap for Speech Translationcited by 15
  6. 2023Recent Advances in Direct Speech-to-text Translationcited by 15
  7. 2024Pushing the Limits of Zero-shot End-to-End Speech Translationcited by 14
  8. 2024LLaST: Improved End-to-end Speech Translation System Leveraged by Large Language Modelscited by 12
  9. 2023Back Translation for Speech-to-text Translation Without Transcriptscited by 9
  10. 2023ComSL: A Composite Speech-Language Model for End-to-End Speech-to-Text Translationcited by 9
  11. 2023Tuning Large language model for End-to-end Speech Translationcited by 9
  12. 2023Aspect-Based Sentiment Analysis with Explicit Sentiment Augmentationscited by 7
15 more
  1. 2022Make More of Your Data: Minimal Effort Data Augmentation for Automatic Speech Recognition and Translationcited by 6
  2. 2022SegAugment: Maximizing the Utility of Speech Translation Data with Segmentation-based Augmentationscited by 6
  3. 2023An Empirical Study of Consistency Regularization for End-to-End Speech-to-Text Translationcited by 6
  4. 2022Generating Synthetic Speech from SpokenVocab for Speech Translationcited by 5
  5. 2022Joint Speech Translation and Named Entity Recognitioncited by 5
  6. 2023A Unified Framework for Generative Data Augmentation: A Comprehensive Surveycited by 4
  7. 2025Prepending or Cross-Attention for Speech-to-Text? An Empirical Comparisoncited by 4
  8. 2022Ensembling Neural Networks for Improved Prediction and Privacy in Early Diagnosis of Sepsiscited by 3
  9. 2023Towards a Deep Understanding of Multilingual End-to-End Speech Translationcited by 3
  10. 2024Representation Purification for End-to-End Speech Translationcited by 3
  11. 2025Speech Translation Refinement using Large Language Modelscited by 2
  12. 2022Efficient Speech Translation with Pre-trained Modelscited by 1
  13. 2025SpeechT: Findings of the First Mentorship in Speech Translationcited by 1
  14. 2024UniPSDA: Unsupervised Pseudo Semantic Data Augmentation for Zero-Shot Cross-Lingual Natural Language Understandingcited by 0
  15. 2025Interpretable Style Takagi-Sugeno-Kang Fuzzy Clusteringcited by 0