mlsanityData to 14 Oct 2026

Representation Purification for End-to-End Speech Translation

arXiv 2412.04266 ↗First posted 5 Dec 2024cs.CL cs.SD eess.AScited by 3

Not on the map. Why?

Abstract (arXiv)

Speech-to-text translation (ST) is a cross-modal task that involves converting spoken language into text in a different language. Previous research primarily focused on enhancing speech translation by facilitating knowledge transfer from machine translation, exploring various methods to bridge the gap between speech and text modalities. Despite substantial progress made, factors in speech that are not relevant to translation content, such as timbre and rhythm, often limit the efficiency of knowledge transfer. In this paper, we conceptualize speech representation as a combination of content-agnostic and content-relevant factors. We examine the impact of content-agnostic factors on translation performance through preliminary experiments and observe a significant performance deterioration when content-agnostic perturbations are introduced to speech signals. To address this issue, we propose a Speech Representation Purification with Supervision Enhancement (SRPSE) framework, which excludes the content-agnostic components within speech representations to mitigate their negative impact on ST. Experiments on MuST-C and CoVoST-2 datasets demonstrate that SRPSE significantly improves translation performance across all translation directions in three settings and achieves preeminent performance under a textittranscript-free setting.

Key facts. Cited by 3 papers in the map (data through 14 Oct 2026). Builds on 45 of its 64 references that are in the map. First posted on arXiv 5 Dec 2024.

Builds on

45 of 64 references.

References · 45

  1. 2014Adam: A Method for Stochastic Optimizationcited by 37,240
  2. 2020wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representationscited by 3,905
  3. 2018SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processingcited by 1,989
  4. 2018A Call for Clarity in Reporting BLEU Scorescited by 1,799
  5. 2019fairseq: A Fast, Extensible Toolkit for Sequence Modelingcited by 1,488
  6. 2019Common Voice: A Massively-Multilingual Speech Corpuscited by 1,384
  7. 2019SpecAugment: A Simple Data Augmentation Method for Automatic Speech Recognitioncited by 1,380
  8. 2020On Layer Normalization in the Transformer Architecturecited by 922
  9. 2020CLUB: A Contrastive Log-ratio Upper Bound of Mutual Informationcited by 235
  10. 2020fairseq S2T: Fast Speech-to-Text Modeling with fairseqcited by 176
  11. 2018Towards End-to-End Prosody Transfer for Expressive Speech Synthesis with Tacotroncited by 173
  12. 2020Unsupervised Speech Decomposition via Triple Information Bottleneckcited by 88
33 more
  1. 2021TorchAudio: Building Blocks for Audio and Speech Processingcited by 83
  2. 2019One-shot Voice Conversion by Separating Speaker and Content Representations with Instance Normalizationcited by 82
  3. 2020ESPnet-ST: All-in-One Speech Translation Toolkitcited by 82
  4. 2020CoVoST 2 and Massively Multilingual Speech-to-Text Translationcited by 74
  5. 2022Cross-modal Contrastive Learning for Speech Translationcited by 61
  6. 2019End-to-End Speech Translation with Knowledge Distillationcited by 58
  7. 2022STEMM: Self-learning with Speech-text Manifold Mixup for Speech Translationcited by 57
  8. 2020Bridging the Modality Gap for Speech-to-Text Translationcited by 55
  9. 2021Improving Speech Translation by Understanding and Learning from the Auxiliary Text Translation Taskcited by 54
  10. 2021Learning Shared Semantic Space for Speech-to-Text Translationcited by 49
  11. 2021Stacked Acoustic-and-Textual Encoding: Integrating the Pre-trained Models into Speech Translation Encoderscited by 48
  12. 2021End-to-end Speech Translation via Cross-modal Progressive Trainingcited by 47
  13. 2019Bridging the Gap between Pre-Training and Fine-Tuning for End-to-End Speech Translationcited by 39
  14. 2020Self-Training for End-to-End Speech Translationcited by 36
  15. 2022SpeechSplit 2.0: Unsupervised speech disentanglement for voice conversion Without tuning autoencoder Bottleneckscited by 28
  16. 2022Sample, Translate, Recombine: Leveraging Audio Alignments for Data Augmentation in End-to-end Speech Translationcited by 27
  17. 2022Revisiting End-to-End Speech-to-Text Translation From Scratchcited by 26
  18. 2023Pre-training for Speech Translation: CTC Meets Optimal Transportcited by 25
  19. 2023CMOT: Cross-modal Mixup via Optimal Transport for Speech Translationcited by 22
  20. 2017Neural Lattice-to-Sequence Models for Uncertain Inputscited by 20
  21. 2021STYLER: Style Factor Modeling with Rapidity and Robustness via Speech Decomposition for Expressive and Controllable Neural Text to Speechcited by 18
  22. 2023DUB: Discrete Unit Back-translation for Speech Translationcited by 18
  23. 2021Regularizing End-to-End Speech Translation with Triangular Decomposition Agreementcited by 16
  24. 2022Speech Representation Disentanglement with Adversarial Mutual Information Learning for One-shot Voice Conversioncited by 14
  25. 2019Self-Attentional Models for Lattice Inputscited by 12
  26. 2020Jointly Trained Transformers models for Spoken Language Translationcited by 10
  27. 2023Co-Salient Object Detection with Co-Representation Purificationcited by 9
  28. 2023Back Translation for Speech-to-text Translation Without Transcriptscited by 9
  29. 2022Content-Context Factorized Representations for Automated Speech Recognitioncited by 8
  30. 2022A Unified Neural Network Model for Readability Assessment with Feature Projection and Length-Balanced Losscited by 7
  31. 2023Soft Alignment of Modality Space for End-to-end Speech Translationcited by 4
  32. 2023Decouple Non-parametric Knowledge Distillation For End-to-end Speech Translationcited by 3
  33. 2023Cross-Modal Multi-Tasking for Speech-to-Text Translation via Hard Parameter Sharingcited by 3

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 2; 2026: 1.

3 papers.

  1. 2025DTW-Align: Bridging the Modality Gap in End-to-End Speech Translation with Dynamic Time Warping Alignmentcited by 2
  2. 2025Emphasis Sensitivity in Speech Representationscited by 0
  3. 2026Cross-Modal Robustness Transfer (CMRT): Training Robust Speech Translation Models Using Adversarial Textcited by 0