mlsanityData to 7 Oct 2026

SegAugment: Maximizing the Utility of Speech Translation Data with Segmentation-based Augmentations

arXiv 2212.09699 ↗First posted 19 Dec 2022cs.CL cs.SD eess.AScited by 6

Not on the map. Why?

Abstract (arXiv)

End-to-end Speech Translation is hindered by a lack of available data resources. While most of them are based on documents, a sentence-level version is available, which is however single and static, potentially impeding the usefulness of the data. We propose a new data augmentation strategy, SegAugment, to address this issue by generating multiple alternative sentence-level versions of a dataset. Our method utilizes an Audio Segmentation system, which re-segments the speech of each document with different length constraints, after which we obtain the target text via alignment methods. Experiments demonstrate consistent gains across eight language pairs in MuST-C, with an average increase of 2.5 BLEU points, and up to 5 BLEU for low-resource scenarios in mTEDx. Furthermore, when combined with a strong system, SegAugment establishes new state-of-the-art results in MuST-C. Finally, we show that the proposed method can also successfully augment sentence-level datasets, and that it enables Speech Translation models to close the gap between the manual and automatic segmentation at inference time.

Key facts. Cited by 6 papers in the map (data through 7 Oct 2026). Builds on 46 of its 76 references that are in the map. First posted on arXiv 19 Dec 2022.

Builds on

46 of 76 references.

References · 46

  1. 2017Attention Is All You Needcited by 59,932
  2. 2020wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representationscited by 3,853
  3. 2016Gaussian Error Linear Units (GELUs)cited by 3,431
  4. 2018SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processingcited by 1,973
  5. 2018A Call for Clarity in Reporting BLEU Scorescited by 1,790
  6. 2019fairseq: A Fast, Extensible Toolkit for Sequence Modelingcited by 1,486
  7. 2019Common Voice: A Massively-Multilingual Speech Corpuscited by 1,358
  8. 2019SpecAugment: A Simple Data Augmentation Method for Automatic Speech Recognitioncited by 1,355
  9. 2020On Layer Normalization in the Transformer Architecturecited by 906
  10. 2016Language Modeling with Gated Convolutional Networkscited by 688
  11. 2021XLS-R: Self-supervised Cross-lingual Speech Representation Learning at Scalecited by 589
  12. 2019Libri-Light: A Benchmark for ASR with Limited or No Supervisioncited by 462
34 more
  1. 2020Multilingual Translation with Extensible Multilingual Pretraining and Finetuningcited by 287
  2. 2020fairseq S2T: Fast Speech-to-Text Modeling with fairseqcited by 174
  3. 2017Sequence-to-Sequence Models Can Directly Translate Foreign Speechcited by 121
  4. 2021The Multilingual TEDx Corpus for Speech Recognition and Translationcited by 101
  5. 2018Context-Aware Neural Machine Translation Learns Anaphora Resolutioncited by 97
  6. 2020Speech Translation and the End-to-End Promise: Taking Stock of Where We Arecited by 77
  7. 2020CTC-Segmentation of Large Corpora for German End-to-end Speech Recognitioncited by 75
  8. 2022Unified Speech-Text Pre-training for Speech Translation and Recognitioncited by 64
  9. 2022Cross-modal Contrastive Learning for Speech Translationcited by 61
  10. 2019End-to-End Speech Translation with Knowledge Distillationcited by 58
  11. 2018Leveraging Weakly Supervised Data to Improve End-to-End Speech-to-Text Translationcited by 57
  12. 2020CoVoST: A Diverse Multilingual Speech-To-Text Translation Corpuscited by 57
  13. 2022STEMM: Self-learning with Speech-text Manifold Mixup for Speech Translationcited by 57
  14. 2021Cascade versus Direct Speech Translation: Do the Differences Still Make a Difference?cited by 54
  15. 2021Learning Shared Semantic Space for Speech-to-Text Translationcited by 49
  16. 2022SpeechUT: Bridging Speech and Text with Hidden-Unit for Encoder-Decoder Based Speech-Text Pre-trainingcited by 40
  17. 2020End-to-End Speech-Translation with Knowledge Distillation: FBK@IWSLT2020cited by 31
  18. 2019Improving sequence-to-sequence speech recognition training with on-the-fly data augmentationcited by 29
  19. 2022Sample, Translate, Recombine: Leveraging Audio Alignments for Data Augmentation in End-to-end Speech Translationcited by 27
  20. 2019Domain, Translationese and Noise in Synthetic Data for Neural Machine Translationcited by 26
  21. 2022SHAS: Approaching optimal Segmentation for End-to-End Speech Translationcited by 26
  22. 2022Revisiting End-to-End Speech-to-Text Translation From Scratchcited by 26
  23. 2020On the Usefulness of Self-Attention for Automatic Speech Recognition with Transformerscited by 24
  24. 2019On Using SpecAugment for End-to-End Speech Translationcited by 19
  25. 2021Beyond Voice Activity Detection: Hybrid Audio Segmentation for Direct Speech Translationcited by 17
  26. 2022The Curious Case of Absolute Position Embeddingscited by 17
  27. 2021End-to-End Speech Translation with Pre-trained Models and Adapters: UPC at IWSLT 2021cited by 16
  28. 2020SkinAugment: Auto-Encoding Speaker Conversions for Automatic Speech Translationcited by 13
  29. 2020Contextualized Translation of Automatically Segmented Speechcited by 12
  30. 2021Data Augmentation by Concatenation for Low-Resource Translation: A Mystery and a Solutioncited by 11
  31. 2021Sentence Concatenation Approach to Data Augmentation for Neural Machine Translationcited by 8
  32. 2022Don't Discard Fixed-Window Audio Segmentation in Speech-to-Text Translationcited by 8
  33. 2021Dealing with training and test segmentation mismatch: FBK@IWSLT2021cited by 6
  34. 2022Make More of Your Data: Minimal Effort Data Augmentation for Automatic Speech Recognition and Translationcited by 6

Built on by

Cited by year

Citing papers per year: 2023: 2; 2024: 4.

6 papers.

  1. 2024StreamAtt: Direct Streaming Speech-to-Text Translation with Attention-based Audio History Selectioncited by 19
  2. 2023Recent Advances in Direct Speech-to-text Translationcited by 15
  3. 2024Pushing the Limits of Zero-shot End-to-End Speech Translationcited by 14
  4. 2024LLaST: Improved End-to-end Speech Translation System Leveraged by Large Language Modelscited by 12
  5. 2023Speech Translation with Foundation Models and Optimal Transport: UPC at IWSLT23cited by 1
  6. 2024Lightweight Audio Segmentation for Long-form Speech Translationcited by 1