# Representation Purification for End-to-End Speech Translation

arXiv: https://arxiv.org/abs/2412.04266 · HTML: https://mlsanity.com/paper/2412.04266/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 45 of its 64 references that are in the map.
- First posted on arXiv 5 Dec 2024.

## Builds on (most cited first, 30 of 45)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representations](https://mlsanity.com/paper/2006.11477/) (2020)
- [SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processing](https://mlsanity.com/paper/1808.06226/) (2018)
- [A Call for Clarity in Reporting BLEU Scores](https://mlsanity.com/paper/1804.08771/) (2018)
- [fairseq: A Fast, Extensible Toolkit for Sequence Modeling](https://mlsanity.com/paper/1904.01038/) (2019)
- [Common Voice: A Massively-Multilingual Speech Corpus](https://mlsanity.com/paper/1912.06670/) (2019)
- [SpecAugment: A Simple Data Augmentation Method for Automatic Speech Recognition](https://mlsanity.com/paper/1904.08779/) (2019)
- [On Layer Normalization in the Transformer Architecture](https://mlsanity.com/paper/2002.04745/) (2020)
- [CLUB: A Contrastive Log-ratio Upper Bound of Mutual Information](https://mlsanity.com/paper/2006.12013/) (2020)
- [fairseq S2T: Fast Speech-to-Text Modeling with fairseq](https://mlsanity.com/paper/2010.05171/) (2020)
- [Towards End-to-End Prosody Transfer for Expressive Speech Synthesis with Tacotron](https://mlsanity.com/paper/1803.09047/) (2018)
- [Unsupervised Speech Decomposition via Triple Information Bottleneck](https://mlsanity.com/paper/2004.11284/) (2020)
- [TorchAudio: Building Blocks for Audio and Speech Processing](https://mlsanity.com/paper/2110.15018/) (2021)
- [One-shot Voice Conversion by Separating Speaker and Content Representations with Instance Normalization](https://mlsanity.com/paper/1904.05742/) (2019)
- [ESPnet-ST: All-in-One Speech Translation Toolkit](https://mlsanity.com/paper/2004.10234/) (2020)
- [CoVoST 2 and Massively Multilingual Speech-to-Text Translation](https://mlsanity.com/paper/2007.10310/) (2020)
- [Cross-modal Contrastive Learning for Speech Translation](https://mlsanity.com/paper/2205.02444/) (2022)
- [End-to-End Speech Translation with Knowledge Distillation](https://mlsanity.com/paper/1904.08075/) (2019)
- [STEMM: Self-learning with Speech-text Manifold Mixup for Speech Translation](https://mlsanity.com/paper/2203.10426/) (2022)
- [Bridging the Modality Gap for Speech-to-Text Translation](https://mlsanity.com/paper/2010.14920/) (2020)
- [Improving Speech Translation by Understanding and Learning from the Auxiliary Text Translation Task](https://mlsanity.com/paper/2107.05782/) (2021)
- [Learning Shared Semantic Space for Speech-to-Text Translation](https://mlsanity.com/paper/2105.03095/) (2021)
- [Stacked Acoustic-and-Textual Encoding: Integrating the Pre-trained Models into Speech Translation Encoders](https://mlsanity.com/paper/2105.05752/) (2021)
- [End-to-end Speech Translation via Cross-modal Progressive Training](https://mlsanity.com/paper/2104.10380/) (2021)
- [Bridging the Gap between Pre-Training and Fine-Tuning for End-to-End Speech Translation](https://mlsanity.com/paper/1909.07575/) (2019)
- [Self-Training for End-to-End Speech Translation](https://mlsanity.com/paper/2006.02490/) (2020)
- [SpeechSplit 2.0: Unsupervised speech disentanglement for voice conversion Without tuning autoencoder Bottlenecks](https://mlsanity.com/paper/2203.14156/) (2022)
- [Sample, Translate, Recombine: Leveraging Audio Alignments for Data Augmentation in End-to-end Speech Translation](https://mlsanity.com/paper/2203.08757/) (2022)
- [Revisiting End-to-End Speech-to-Text Translation From Scratch](https://mlsanity.com/paper/2206.04571/) (2022)
- [Pre-training for Speech Translation: CTC Meets Optimal Transport](https://mlsanity.com/paper/2301.11716/) (2023)

## Built on by (most cited first, 3 of 3)

- [DTW-Align: Bridging the Modality Gap in End-to-End Speech Translation with Dynamic Time Warping Alignment](https://mlsanity.com/paper/2509.18987/) (2025)
- [Emphasis Sensitivity in Speech Representations](https://mlsanity.com/paper/2508.11566/) (2025)
- [Cross-Modal Robustness Transfer (CMRT): Training Robust Speech Translation Models Using Adversarial Text](https://mlsanity.com/paper/2602.11933/) (2026)

## Abstract (arXiv)

Speech-to-text translation (ST) is a cross-modal task that involves converting spoken language into text in a different language. Previous research primarily focused on enhancing speech translation by facilitating knowledge transfer from machine translation, exploring various methods to bridge the gap between speech and text modalities. Despite substantial progress made, factors in speech that are not relevant to translation content, such as timbre and rhythm, often limit the efficiency of knowledge transfer. In this paper, we conceptualize speech representation as a combination of content-agnostic and content-relevant factors. We examine the impact of content-agnostic factors on translation performance through preliminary experiments and observe a significant performance deterioration when content-agnostic perturbations are introduced to speech signals. To address this issue, we propose a Speech Representation Purification with Supervision Enhancement (SRPSE) framework, which excludes the content-agnostic components within speech representations to mitigate their negative impact on ST. Experiments on MuST-C and CoVoST-2 datasets demonstrate that SRPSE significantly improves translation performance across all translation directions in three settings and achieves preeminent performance under a textittranscript-free setting.