# Adaptive Feature Selection for End-to-End Speech Translation

arXiv: https://arxiv.org/abs/2010.08518 · HTML: https://mlsanity.com/paper/2010.08518/

## Key facts

- Cited by 25 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 16 Oct 2020.

## Built on by (most cited first, 25 of 25)

- [STEMM: Self-learning with Speech-text Manifold Mixup for Speech Translation](https://mlsanity.com/paper/2203.10426/) (2022)
- [Learning Shared Semantic Space for Speech-to-Text Translation](https://mlsanity.com/paper/2105.03095/) (2021)
- [End-to-end Speech Translation via Cross-modal Progressive Training](https://mlsanity.com/paper/2104.10380/) (2021)
- [RealTranS: End-to-End Simultaneous Speech Translation with Convolutional Weighted-Shrinking Transformer](https://mlsanity.com/paper/2106.04833/) (2021)
- [Information-Transport-based Policy for Simultaneous Translation](https://mlsanity.com/paper/2210.12357/) (2022)
- [Learning When to Translate for Streaming Speech](https://mlsanity.com/paper/2109.07368/) (2021)
- [Revisiting End-to-End Speech-to-Text Translation From Scratch](https://mlsanity.com/paper/2206.04571/) (2022)
- [Pre-training for Speech Translation: CTC Meets Optimal Transport](https://mlsanity.com/paper/2301.11716/) (2023)
- [Accelerating RNN-T Training and Inference Using CTC guidance](https://mlsanity.com/paper/2210.16481/) (2022)
- [Regularizing End-to-End Speech Translation with Triangular Decomposition Agreement](https://mlsanity.com/paper/2112.10991/) (2021)
- [Speechformer: Reducing Information Loss in Direct Speech Translation](https://mlsanity.com/paper/2109.04574/) (2021)
- [M-Adapter: Modality Adaptation for End-to-End Speech-to-Text Translation](https://mlsanity.com/paper/2207.00952/) (2022)
- [Non-Parametric Domain Adaptation for End-to-End Speech Translation](https://mlsanity.com/paper/2205.11211/) (2022)
- [Fast-MD: Fast Multi-Decoder End-to-End Speech Translation with Non-Autoregressive Hidden Intermediates](https://mlsanity.com/paper/2109.12804/) (2021)
- [Improving Speech Translation by Cross-Modal Multi-Grained Contrastive Learning](https://mlsanity.com/paper/2304.10309/) (2023)
- [Discrete Cross-Modal Alignment Enables Zero-Shot Speech Translation](https://mlsanity.com/paper/2210.09556/) (2022)
- [Towards a Deep Understanding of Multilingual End-to-End Speech Translation](https://mlsanity.com/paper/2310.20456/) (2023)
- [RedApt: An Adaptor for wav2vec 2 Encoding  Faster and Smaller Speech Translation without Quality Compromise](https://mlsanity.com/paper/2210.08475/) (2022)
- [Bridging the Granularity Gap for Acoustic Modeling](https://mlsanity.com/paper/2305.17356/) (2023)
- [Multiformer: A Head-Configurable Transformer-Based Model for Direct Speech Translation](https://mlsanity.com/paper/2205.07100/) (2022)
- [Efficient CTC Regularization via Coarse Labels for End-to-End Speech Translation](https://mlsanity.com/paper/2302.10871/) (2023)
- [Speech Translation with Foundation Models and Optimal Transport: UPC at IWSLT23](https://mlsanity.com/paper/2306.01327/) (2023)
- [How do Hyenas deal with Human Speech? Speech Recognition and Translation with ConfHyena](https://mlsanity.com/paper/2402.13208/) (2024)
- [JoeyS2T: Minimalistic Speech-to-Text Modeling with JoeyNMT](https://mlsanity.com/paper/2210.02545/) (2022)
- [Joint Training And Decoding for Multilingual End-to-End Simultaneous Speech Translation](https://mlsanity.com/paper/2503.11080/) (2025)

## Abstract (arXiv)

Information in speech signals is not evenly distributed, making it an additional challenge for end-to-end (E2E) speech translation (ST) to learn to focus on informative features. In this paper, we propose adaptive feature selection (AFS) for encoder-decoder based E2E ST. We first pre-train an ASR encoder and apply AFS to dynamically estimate the importance of each encoded speech feature to SR. A ST encoder, stacked on top of the ASR encoder, then receives the filtered features from the (frozen) ASR encoder. We take L0DROP (Zhang et al., 2020) as the backbone for AFS, and adapt it to sparsify speech features with respect to both temporal and feature dimensions. Results on LibriSpeech En-Fr and MuST-C benchmarks show that AFS facilitates learning of ST by pruning out ~84% temporal features, yielding an average translation gain of ~1.3-1.6 BLEU and a decoding speedup of ~1.4x. In particular, AFS reduces the performance gap compared to the cascade baseline, and outperforms it on LibriSpeech En-Fr with a BLEU score of 18.56 (without data augmentation)