mlsanityData to 7 Oct 2026

Large-Scale Streaming End-to-End Speech Translation with Neural Transducers

arXiv 2204.05352 ↗First posted 11 Apr 2022cs.CL eess.AScited by 31

Not on the map. Why?

Abstract (arXiv)

Neural transducers have been widely used in automatic speech recognition (ASR). In this paper, we introduce it to streaming end-to-end speech translation (ST), which aims to convert audio signals to texts in other languages directly. Compared with cascaded ST that performs ASR followed by text-based machine translation (MT), the proposed Transformer transducer (TT)-based ST model drastically reduces inference latency, exploits speech information, and avoids error propagation from ASR to MT. To improve the modeling capacity, we propose attention pooling for the joint network in TT. In addition, we extend TT-based ST to multilingual ST, which generates texts of multiple languages at the same time. Experimental results on a large-scale 50 thousand (K) hours pseudo-labeled training set show that TT-based ST not only significantly reduces inference time but also outperforms non-streaming cascaded ST for English-German translation.

Key facts. Cited by 31 papers in the map (data through 7 Oct 2026). Builds on 22 of its 34 references that are in the map. First posted on arXiv 11 Apr 2022.

Builds on

22 of 34 references.

References · 22

  1. 2020Transformer Transducer: A Streamable Speech Recognition Model with Transformer Encoders and RNN-T Losscited by 232
  2. 2021Recent Advances in End-to-End Automatic Speech Recognitioncited by 138
  3. 2017Sequence-to-Sequence Models Can Directly Translate Foreign Speechcited by 121
  4. 2016Listen and Translate: A Proof of Concept for End-to-End Speech-to-Text Translationcited by 113
  5. 2019Monotonic Infinite Lookback Attention for Simultaneous Machine Translationcited by 103
  6. 2020A Streaming On-Device End-to-End Model Surpassing Server-Side Conventional Model Quality and Latencycited by 99
  7. 2020SimulEval: An Evaluation Toolkit for Simultaneous Translationcited by 94
  8. 2020Developing Real-time Streaming Transformer Transducer for Speech Recognition on Large-scale Datasetcited by 94
  9. 2019Monotonic Multihead Attentioncited by 91
  10. 2017Online and Linear-Time Attention by Enforcing Monotonic Alignmentscited by 85
  11. 2017Monotonic Chunkwise Attentioncited by 82
  12. 2020Speech Translation and the End-to-End Promise: Taking Stock of Where We Arecited by 77
10 more
  1. 2019Transformer-Transducer: End-to-End Speech Recognition with Self-Attentioncited by 68
  2. 2018End-to-End Automatic Speech Translation of Audiobookscited by 65
  3. 2020Developing RNN-T Models Surpassing High-Performance Hybrid Models with Customization Capabilitycited by 63
  4. 2019End-to-End Speech Translation with Knowledge Distillationcited by 58
  5. 2018Leveraging Weakly Supervised Data to Improve End-to-End Speech-to-Text Translationcited by 57
  6. 2021Advancing RNN Transducer Technology for Speech Recognitioncited by 44
  7. 2020Streaming Simultaneous Speech Translation with Augmented Memory Transformercited by 30
  8. 2020Minimum Latency Training Strategies for Streaming Sequence-to-Sequence ASRcited by 19
  9. 2020Multitask Training with Text Data for End-to-End Speech Recognitioncited by 19
  10. 2022Improving the fusion of acoustic and text representations in RNN-Tcited by 11

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 2; 2023: 9; 2024: 8; 2025: 9; 2026: 3.

31 papers.

Sort:

By area:

  1. 2023On decoder-only architecture for speech-to-text and large language model integrationcited by 144
  2. 2023Efficient Sequence Transduction by Jointly Predicting Tokens and Durationscited by 56
  3. 2023Hybrid Transducer and Attention based Encoder-Decoder Modeling for Speech-to-Text Taskscited by 20
  4. 2024How "Real" is Your Real-Time Simultaneous Speech-to-Text Translation System?cited by 17
  5. 2024Paralinguistics-Aware Speech-Empowered Large Language Models for Natural Conversationcited by 14
  6. 2022A Weakly-Supervised Streaming Multilingual Speech Model with Truly Zero-Shot Capabilitycited by 12
  7. 2022LAMASSU: Streaming Language-Agnostic Multilingual Speech Recognition and Translation Using Neural Transducerscited by 10
  8. 2023ESPnet-ST-v2: Multipurpose Spoken Language Translation Toolkitcited by 10
  9. 2023End-to-End Speech-to-Text Translation: A Surveycited by 9
  10. 2023Token-Level Serialized Output Training for Joint Streaming ASR and ST Leveraging Textual Alignmentscited by 6
  11. 2023DiariST: Streaming Speech Translation with Speaker Diarizationcited by 5
  12. 2024Label-Synchronous Neural Transducer for E2E Simultaneous Speech Translationcited by 5
19 more
  1. 2024Overcoming Non-monotonicity in Transducer-based Streaming Generationcited by 4
  2. 2025SimulS2S-LLM: Unlocking Simultaneous Inference of Speech LLMs for Speech-to-Speech Translationcited by 3
  3. 2025Spatial Speech Translation: Translating Across Space With Binaural Hearablescited by 3
  4. 2023Improving Stability in Simultaneous Speech Translation: A Revision-Controllable Decoding Approachcited by 2
  5. 2023End-to-End Single-Channel Speaker-Turn Aware Conversational Speech Translationcited by 2
  6. 2024Streaming Sequence Transduction through Dynamic Compressioncited by 2
  7. 2025SimulMEGA: MoE Routers are Advanced Policy Makers for Simultaneous Speech Translationcited by 2
  8. 2024CTC-GMM: CTC guided modality matching for fast and accurate streaming speech translationcited by 1
  9. 2024Transducer Consistency Regularization for Speech to Text Applicationscited by 1
  10. 2025Aligner-Encoders: Self-Attention Transformers Can Be Self-Transducerscited by 1
  11. 2025Length Aware Speech Translation for Video Dubbingcited by 1
  12. 2025HENT-SRT: Hierarchical Efficient Neural Transducer with Self-Distillation for Joint Speech Recognition and Translationcited by 1
  13. 2025PHRASED: Phrase Dictionary Biasing for Speech Translationcited by 1
  14. 2025REINA: Regularized Entropy Information-Based Loss for Efficient Simultaneous Speech Translationcited by 1
  15. 2024Soft Language Identification for Language-Agnostic Many-to-One End-to-End Speech Translationcited by 0
  16. 2025Joint Training And Decoding for Multilingual End-to-End Simultaneous Speech Translationcited by 0
  17. 2026Chunk-wise Attention Transducers for Fast and Accurate Streaming Speech-to-Textcited by 0
  18. 2026Regularized Entropy Information Adaptation with Temporal-Awareness Networks for Simultaneous Speech Translationcited by 0
  19. 2026Do LLMs Need Architectural Changes for Simultaneous Speech Translation? A Prefix-to-Prefix Data Driven Approachcited by 0