# Large-Scale Streaming End-to-End Speech Translation with Neural Transducers

arXiv: https://arxiv.org/abs/2204.05352 · HTML: https://mlsanity.com/paper/2204.05352/

## Key facts

- Cited by 31 papers in the map (data through 14 Oct 2026).
- Builds on 22 of its 34 references that are in the map.
- First posted on arXiv 11 Apr 2022.

## Builds on (most cited first, 22 of 22)

- [Transformer Transducer: A Streamable Speech Recognition Model with Transformer Encoders and RNN-T Loss](https://mlsanity.com/paper/2002.02562/) (2020)
- [Recent Advances in End-to-End Automatic Speech Recognition](https://mlsanity.com/paper/2111.01690/) (2021)
- [Sequence-to-Sequence Models Can Directly Translate Foreign Speech](https://mlsanity.com/paper/1703.08581/) (2017)
- [Listen and Translate: A Proof of Concept for End-to-End Speech-to-Text Translation](https://mlsanity.com/paper/1612.01744/) (2016)
- [Monotonic Infinite Lookback Attention for Simultaneous Machine Translation](https://mlsanity.com/paper/1906.05218/) (2019)
- [A Streaming On-Device End-to-End Model Surpassing Server-Side Conventional Model Quality and Latency](https://mlsanity.com/paper/2003.12710/) (2020)
- [Developing Real-time Streaming Transformer Transducer for Speech Recognition on Large-scale Dataset](https://mlsanity.com/paper/2010.11395/) (2020)
- [SimulEval: An Evaluation Toolkit for Simultaneous Translation](https://mlsanity.com/paper/2007.16193/) (2020)
- [Monotonic Multihead Attention](https://mlsanity.com/paper/1909.12406/) (2019)
- [Online and Linear-Time Attention by Enforcing Monotonic Alignments](https://mlsanity.com/paper/1704.00784/) (2017)
- [Monotonic Chunkwise Attention](https://mlsanity.com/paper/1712.05382/) (2017)
- [Speech Translation and the End-to-End Promise: Taking Stock of Where We Are](https://mlsanity.com/paper/2004.06358/) (2020)
- [Transformer-Transducer: End-to-End Speech Recognition with Self-Attention](https://mlsanity.com/paper/1910.12977/) (2019)
- [End-to-End Automatic Speech Translation of Audiobooks](https://mlsanity.com/paper/1802.04200/) (2018)
- [Developing RNN-T Models Surpassing High-Performance Hybrid Models with Customization Capability](https://mlsanity.com/paper/2007.15188/) (2020)
- [End-to-End Speech Translation with Knowledge Distillation](https://mlsanity.com/paper/1904.08075/) (2019)
- [Leveraging Weakly Supervised Data to Improve End-to-End Speech-to-Text Translation](https://mlsanity.com/paper/1811.02050/) (2018)
- [Advancing RNN Transducer Technology for Speech Recognition](https://mlsanity.com/paper/2103.09935/) (2021)
- [Streaming Simultaneous Speech Translation with Augmented Memory Transformer](https://mlsanity.com/paper/2011.00033/) (2020)
- [Minimum Latency Training Strategies for Streaming Sequence-to-Sequence ASR](https://mlsanity.com/paper/2004.05009/) (2020)
- [Multitask Training with Text Data for End-to-End Speech Recognition](https://mlsanity.com/paper/2010.14318/) (2020)
- [Improving the fusion of acoustic and text representations in RNN-T](https://mlsanity.com/paper/2201.10240/) (2022)

## Built on by (most cited first, 30 of 31)

- [On decoder-only architecture for speech-to-text and large language model integration](https://mlsanity.com/paper/2307.03917/) (2023)
- [Efficient Sequence Transduction by Jointly Predicting Tokens and Durations](https://mlsanity.com/paper/2304.06795/) (2023)
- [Hybrid Transducer and Attention based Encoder-Decoder Modeling for Speech-to-Text Tasks](https://mlsanity.com/paper/2305.03101/) (2023)
- [How "Real" is Your Real-Time Simultaneous Speech-to-Text Translation System?](https://mlsanity.com/paper/2412.18495/) (2024)
- [Paralinguistics-Aware Speech-Empowered Large Language Models for Natural Conversation](https://mlsanity.com/paper/2402.05706/) (2024)
- [A Weakly-Supervised Streaming Multilingual Speech Model with Truly Zero-Shot Capability](https://mlsanity.com/paper/2211.02499/) (2022)
- [LAMASSU: Streaming Language-Agnostic Multilingual Speech Recognition and Translation Using Neural Transducers](https://mlsanity.com/paper/2211.02809/) (2022)
- [ESPnet-ST-v2: Multipurpose Spoken Language Translation Toolkit](https://mlsanity.com/paper/2304.04596/) (2023)
- [End-to-End Speech-to-Text Translation: A Survey](https://mlsanity.com/paper/2312.01053/) (2023)
- [Token-Level Serialized Output Training for Joint Streaming ASR and ST Leveraging Textual Alignments](https://mlsanity.com/paper/2307.03354/) (2023)
- [DiariST: Streaming Speech Translation with Speaker Diarization](https://mlsanity.com/paper/2309.08007/) (2023)
- [Label-Synchronous Neural Transducer for E2E Simultaneous Speech Translation](https://mlsanity.com/paper/2406.04541/) (2024)
- [Overcoming Non-monotonicity in Transducer-based Streaming Generation](https://mlsanity.com/paper/2411.17170/) (2024)
- [SimulS2S-LLM: Unlocking Simultaneous Inference of Speech LLMs for Speech-to-Speech Translation](https://mlsanity.com/paper/2504.15509/) (2025)
- [Spatial Speech Translation: Translating Across Space With Binaural Hearables](https://mlsanity.com/paper/2504.18715/) (2025)
- [Improving Stability in Simultaneous Speech Translation: A Revision-Controllable Decoding Approach](https://mlsanity.com/paper/2310.04399/) (2023)
- [End-to-End Single-Channel Speaker-Turn Aware Conversational Speech Translation](https://mlsanity.com/paper/2311.00697/) (2023)
- [Streaming Sequence Transduction through Dynamic Compression](https://mlsanity.com/paper/2402.01172/) (2024)
- [SimulMEGA: MoE Routers are Advanced Policy Makers for Simultaneous Speech Translation](https://mlsanity.com/paper/2509.01200/) (2025)
- [CTC-GMM: CTC guided modality matching for fast and accurate streaming speech translation](https://mlsanity.com/paper/2410.05146/) (2024)
- [Transducer Consistency Regularization for Speech to Text Applications](https://mlsanity.com/paper/2410.07491/) (2024)
- [Aligner-Encoders: Self-Attention Transformers Can Be Self-Transducers](https://mlsanity.com/paper/2502.05232/) (2025)
- [Length Aware Speech Translation for Video Dubbing](https://mlsanity.com/paper/2506.00740/) (2025)
- [HENT-SRT: Hierarchical Efficient Neural Transducer with Self-Distillation for Joint Speech Recognition and Translation](https://mlsanity.com/paper/2506.02157/) (2025)
- [PHRASED: Phrase Dictionary Biasing for Speech Translation](https://mlsanity.com/paper/2506.09175/) (2025)
- [REINA: Regularized Entropy Information-Based Loss for Efficient Simultaneous Speech Translation](https://mlsanity.com/paper/2508.04946/) (2025)
- [Soft Language Identification for Language-Agnostic Many-to-One End-to-End Speech Translation](https://mlsanity.com/paper/2406.10276/) (2024)
- [Joint Training And Decoding for Multilingual End-to-End Simultaneous Speech Translation](https://mlsanity.com/paper/2503.11080/) (2025)
- [Chunk-wise Attention Transducers for Fast and Accurate Streaming Speech-to-Text](https://mlsanity.com/paper/2602.24245/) (2026)
- [Regularized Entropy Information Adaptation with Temporal-Awareness Networks for Simultaneous Speech Translation](https://mlsanity.com/paper/2604.09916/) (2026)

## Abstract (arXiv)

Neural transducers have been widely used in automatic speech recognition (ASR). In this paper, we introduce it to streaming end-to-end speech translation (ST), which aims to convert audio signals to texts in other languages directly. Compared with cascaded ST that performs ASR followed by text-based machine translation (MT), the proposed Transformer transducer (TT)-based ST model drastically reduces inference latency, exploits speech information, and avoids error propagation from ASR to MT. To improve the modeling capacity, we propose attention pooling for the joint network in TT. In addition, we extend TT-based ST to multilingual ST, which generates texts of multiple languages at the same time. Experimental results on a large-scale 50 thousand (K) hours pseudo-labeled training set show that TT-based ST not only significantly reduces inference time but also outperforms non-streaming cascaded ST for English-German translation.