# Attention-Passing Models for Robust and Data-Efficient End-to-End Speech Translation

arXiv: https://arxiv.org/abs/1904.07209 · HTML: https://mlsanity.com/paper/1904.07209/

## Key facts

- Cited by 30 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 15 Apr 2019.

## Built on by (most cited first, 30 of 30)

- [The Multilingual TEDx Corpus for Speech Recognition and Translation](https://mlsanity.com/paper/2102.01757/) (2021)
- [UnitY: Two-pass Direct Speech-to-speech Translation with Discrete Units](https://mlsanity.com/paper/2212.08055/) (2022)
- [Attention, please! A survey of Neural Attention Models in Deep Learning](https://mlsanity.com/paper/2103.16775/) (2021)
- [Stacked Acoustic-and-Textual Encoding: Integrating the Pre-trained Models into Speech Translation Encoders](https://mlsanity.com/paper/2105.05752/) (2021)
- [Source and Target Bidirectional Knowledge Distillation for End-to-end Speech Translation](https://mlsanity.com/paper/2104.06457/) (2021)
- [Pre-training for Speech Translation: CTC Meets Optimal Transport](https://mlsanity.com/paper/2301.11716/) (2023)
- [Searchable Hidden Intermediates for End-to-End Models of Decomposable Sequence Tasks](https://mlsanity.com/paper/2105.00573/) (2021)
- [Investigating the Reordering Capability in CTC-based Non-Autoregressive End-to-End Speech Translation](https://mlsanity.com/paper/2105.04840/) (2021)
- [BSTC: A Large-Scale Chinese-English Speech Translation Dataset](https://mlsanity.com/paper/2104.03575/) (2021)
- [Regularizing End-to-End Speech Translation with Triangular Decomposition Agreement](https://mlsanity.com/paper/2112.10991/) (2021)
- [Recent Advances in Direct Speech-to-text Translation](https://mlsanity.com/paper/2306.11646/) (2023)
- [Non-Parametric Domain Adaptation for End-to-End Speech Translation](https://mlsanity.com/paper/2205.11211/) (2022)
- [LLaST: Improved End-to-end Speech Translation System Leveraged by Large Language Models](https://mlsanity.com/paper/2407.15415/) (2024)
- [Fast-MD: Fast Multi-Decoder End-to-End Speech Translation with Non-Autoregressive Hidden Intermediates](https://mlsanity.com/paper/2109.12804/) (2021)
- [Tuning Large language model for End-to-end Speech Translation](https://mlsanity.com/paper/2310.02050/) (2023)
- [End-to-End Speech-to-Text Translation: A Survey](https://mlsanity.com/paper/2312.01053/) (2023)
- [4D ASR: Joint modeling of CTC, Attention, Transducer, and Mask-Predict decoders](https://mlsanity.com/paper/2212.10818/) (2022)
- [Improving Speech Translation by Cross-Modal Multi-Grained Contrastive Learning](https://mlsanity.com/paper/2304.10309/) (2023)
- [Does Joint Training Really Help Cascaded Speech Translation?](https://mlsanity.com/paper/2210.13700/) (2022)
- [Align, Write, Re-order: Explainable End-to-End Speech Translation via Operation Sequence Generation](https://mlsanity.com/paper/2211.05967/) (2022)
- [Segmenting Subtitles for Correcting ASR Segmentation Errors](https://mlsanity.com/paper/2104.07868/) (2021)
- [MCAT: Scaling Many-to-Many Speech-to-Text Translation with MLLMs to 70 Languages](https://mlsanity.com/paper/2512.01512/) (2025)
- [AlloST: Low-resource Speech Translation without Source Transcription](https://mlsanity.com/paper/2105.00171/) (2021)
- [Improving Stability in Simultaneous Speech Translation: A Revision-Controllable Decoding Approach](https://mlsanity.com/paper/2310.04399/) (2023)
- [Extremely Simple Out-of-distribution Detection for Audio-visual Generalized Zero-shot Learning](https://mlsanity.com/paper/2503.22197/) (2025)
- [The IWSLT 2021 BUT Speech Translation Systems](https://mlsanity.com/paper/2107.06155/) (2021)
- [Integrated Training for Sequence-to-Sequence Models Using Non-Autoregressive Transformer](https://mlsanity.com/paper/2109.12950/) (2021)
- [JoeyS2T: Minimalistic Speech-to-Text Modeling with JoeyNMT](https://mlsanity.com/paper/2210.02545/) (2022)
- [Objective Soups: Multilingual Multi-Task Modeling for Speech Processing](https://mlsanity.com/paper/2508.09228/) (2025)
- [Listen, Attend, Understand: a Regularization Technique for Stable E2E Speech Translation Training on High Variance labels](https://mlsanity.com/paper/2601.01121/) (2026)

## Abstract (arXiv)

Speech translation has traditionally been approached through cascaded models consisting of a speech recognizer trained on a corpus of transcribed speech, and a machine translation system trained on parallel texts. Several recent works have shown the feasibility of collapsing the cascade into a single, direct model that can be trained in an end-to-end fashion on a corpus of translated speech. However, experiments are inconclusive on whether the cascade or the direct model is stronger, and have only been conducted under the unrealistic assumption that both are trained on equal amounts of data, ignoring other available speech recognition and machine translation corpora. In this paper, we demonstrate that direct speech translation models require more data to perform well than cascaded models, and while they allow including auxiliary data through multi-task training, they are poor at exploiting such data, putting them at a severe disadvantage. As a remedy, we propose the use of end-to-end trainable models with two attention mechanisms, the first establishing source speech to source text alignments, the second modeling source to target text alignment. We show that such models naturally decompose into multi-task-trainable recognition and translation tasks and propose an attention-passing technique that alleviates error propagation issues in a previous formulation of a model with two attention stages. Our proposed model outperforms all examined baselines and is able to exploit auxiliary training data much more effectively than direct attentional models.