# When End-to-End is Overkill: Rethinking Cascaded Speech-to-Text Translation

arXiv: https://arxiv.org/abs/2502.00377 · HTML: https://mlsanity.com/paper/2502.00377/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 26 of its 34 references that are in the map.
- First posted on arXiv 1 Feb 2025.

## Builds on (most cited first, 26 of 26)

- [On the Opportunities and Risks of Foundation Models](https://mlsanity.com/paper/2108.07258/) (2021)
- [HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Units](https://mlsanity.com/paper/2106.07447/) (2021)
- [Multilingual Denoising Pre-training for Neural Machine Translation](https://mlsanity.com/paper/2001.08210/) (2020)
- [SUPERB: Speech processing Universal PERformance Benchmark](https://mlsanity.com/paper/2105.01051/) (2021)
- [VoxPopuli: A Large-Scale Multilingual Speech Corpus for Representation Learning, Semi-Supervised Learning and Interpretation](https://mlsanity.com/paper/2101.00390/) (2021)
- [GigaSpeech: An Evolving, Multi-domain ASR Corpus with 10,000 Hours of Transcribed Audio](https://mlsanity.com/paper/2106.06909/) (2021)
- [Generative Spoken Language Modeling from Raw Audio](https://mlsanity.com/paper/2102.01192/) (2021)
- [Pre-Trained Models: Past, Present and Future](https://mlsanity.com/paper/2106.07139/) (2021)
- [Speech Resynthesis from Discrete Disentangled Self-Supervised Representations](https://mlsanity.com/paper/2104.00355/) (2021)
- [Direct speech-to-speech translation with discrete units](https://mlsanity.com/paper/2107.05604/) (2021)
- [WeNet: Production oriented Streaming and Non-streaming End-to-End Speech Recognition Toolkit](https://mlsanity.com/paper/2102.01547/) (2021)
- [Listen and Translate: A Proof of Concept for End-to-End Speech-to-Text Translation](https://mlsanity.com/paper/1612.01744/) (2016)
- [Leveraging Weakly Supervised Data to Improve End-to-End Speech-to-Text Translation](https://mlsanity.com/paper/1811.02050/) (2018)
- [Enhanced Direct Speech-to-Speech Translation Using Self-supervised Pre-training and Data Augmentation](https://mlsanity.com/paper/2204.02967/) (2022)
- [UWSpeech: Speech to Speech Translation for Unwritten Languages](https://mlsanity.com/paper/2006.07926/) (2020)
- [Self-Training for End-to-End Speech Translation](https://mlsanity.com/paper/2006.02490/) (2020)
- [Large-Scale Self- and Semi-Supervised Learning for Speech Translation](https://mlsanity.com/paper/2104.06678/) (2021)
- [Speech-to-speech Translation between Untranscribed Unknown Languages](https://mlsanity.com/paper/1910.00795/) (2019)
- [Analyzing ASR pretraining for low-resource speech-to-text translation](https://mlsanity.com/paper/1910.10762/) (2019)
- [Searchable Hidden Intermediates for End-to-End Models of Decomposable Sequence Tasks](https://mlsanity.com/paper/2105.00573/) (2021)
- [GigaST: A 10,000-hour Pseudo Speech Translation Corpus](https://mlsanity.com/paper/2204.03939/) (2022)
- [Robust Neural Machine Translation for Clean and Noisy Speech Transcripts](https://mlsanity.com/paper/1910.10238/) (2019)
- [Self-Attentional Models for Lattice Inputs](https://mlsanity.com/paper/1906.01617/) (2019)
- [Fast-MD: Fast Multi-Decoder End-to-End Speech Translation with Non-Autoregressive Hidden Intermediates](https://mlsanity.com/paper/2109.12804/) (2021)
- [Breaking the Data Barrier: Towards Robust Speech Translation via Adversarial Stability Training](https://mlsanity.com/paper/1909.11430/) (2019)
- [Faster Algorithms for Longest Common Substring](https://mlsanity.com/paper/2105.03106/) (2021)

## Built on by (most cited first, 4 of 4)

- [Hearing to Translate: The Effectiveness of Speech Modality Integration into LLMs](https://mlsanity.com/paper/2512.16378/) (2025)
- [Hearing Between the Lines: Unlocking the Reasoning Power of LLMs for Speech Evaluation](https://mlsanity.com/paper/2601.13742/) (2026)
- [LoASR-Bench: Evaluating Large Speech Language Models on Low-Resource Automatic Speech Recognition Across Language Families](https://mlsanity.com/paper/2603.20042/) (2026)
- [It's Not a Walk in the Park! Challenges of Idiom Translation in Speech-to-text Systems](https://mlsanity.com/paper/2506.02995/) (2025)

## Abstract (arXiv)

Though end-to-end speech-to-text translation has been a great success, we argue that the cascaded speech-to-text translation model still has its place, which is usually criticized for the error propagation between automatic speech recognition (ASR) and machine translation (MT) models. In this paper, we explore the benefits of incorporating multiple candidates from ASR and self-supervised speech features into MT. Our analysis reveals that the primary cause of cascading errors stems from the increased divergence between similar samples in the speech domain when mapped to the text domain. By including multiple candidates and self-supervised speech features, our approach allows the machine translation model to choose the right words and ensure precise translation using various speech samples. This strategy minimizes error spread and takes advantage of large ASR and MT datasets, along with pre-trained ASR/MT models, while addressing associated issues.