mlsanityData to 14 Oct 2026

How "Real" is Your Real-Time Simultaneous Speech-to-Text Translation System?

arXiv 2412.18495 ↗First posted 24 Dec 2024cs.CL cs.AI cs.SDcited by 17

Not on the map. Why?

Abstract (arXiv)

Simultaneous speech-to-text translation (SimulST) translates source-language speech into target-language text concurrently with the speaker's speech, ensuring low latency for better user comprehension. Despite its intended application to unbounded speech, most research has focused on human pre-segmented speech, simplifying the task and overlooking significant challenges. This narrow focus, coupled with widespread terminological inconsistencies, is limiting the applicability of research outcomes to real-world applications, ultimately hindering progress in the field. Our extensive literature review of 110 papers not only reveals these critical issues in current research but also serves as the foundation for our key contributions. We 1) define the steps and core components of a SimulST system, proposing a standardized terminology and taxonomy; 2) conduct a thorough analysis of community trends, and 3) offer concrete recommendations and future directions to bridge the gaps in existing literature, from evaluation frameworks to system architectures, for advancing the field towards more realistic and effective SimulST solutions.

Key facts. Cited by 17 papers in the map (data through 14 Oct 2026). Builds on 120 of its 222 references that are in the map. First posted on arXiv 24 Dec 2024.

Builds on

120 of 222 references.

References · 120

  1. 2020Language Models are Few-Shot Learnerscited by 32,419
  2. 2022Robust Speech Recognition via Large-Scale Weak Supervisioncited by 4,057
  3. 2019Transformer-XL: Attentive Language Models Beyond a Fixed-Length Contextcited by 1,850
  4. 2020COMET: A Neural Framework for MT Evaluationcited by 1,033
  5. 2020Efficient Transformers: A Surveycited by 790
  6. 2020Transformer Transducer: A Streamable Speech Recognition Model with Transformer Encoders and RNN-T Losscited by 235
  7. 2023Google USM: Scaling Automatic Speech Recognition Beyond 100 Languagescited by 231
  8. 2023Seamless: Multilingual Expressive and Streaming Speech Translationcited by 192
  9. 2021A Review of Speaker Diarization: Recent Advances with Deep Learningcited by 124
  10. 2017Sequence-to-Sequence Models Can Directly Translate Foreign Speechcited by 121
  11. 2016Listen and Translate: A Proof of Concept for End-to-End Speech-to-Text Translationcited by 113
  12. 2017Neural Machine Translation with Extended Contextcited by 95
38 more
  1. 2015Multi-task Sequence to Sequence Learningcited by 94
  2. 2020SimulEval: An Evaluation Toolkit for Simultaneous Translationcited by 94
  3. 2016Can neural machine translation do simultaneous translation?cited by 86
  4. 2020SimulMT to SimulST: Adapting Simultaneous Text Translation to End-to-End Simultaneous Speech Translationcited by 81
  5. 2019Recognizing long-form speech using streaming end-to-end modelscited by 77
  6. 2020Speech Translation and the End-to-End Promise: Taking Stock of Where We Arecited by 77
  7. 2020Streaming automatic speech recognition with the transformer modelcited by 73
  8. 2018Pre-training on high-resource speech recognition improves low-resource speech-to-text translationcited by 70
  9. 2020Efficient Wait-k Models for Simultaneous Machine Translationcited by 61
  10. 2022Over-Generation Cannot Be Rewarded: Length-Adaptive Average Lagging for Simultaneous Speech Translationcited by 57
  11. 2021Cascade versus Direct Speech Translation: Do the Differences Still Make a Difference?cited by 54
  12. 2018Incremental Decoding and Training Methods for Simultaneous Translation in Neural Machine Translationcited by 52
  13. 2021Measuring and Increasing Context Usage in Context-Aware Machine Translationcited by 48
  14. 2019When and Why is Document-level Context Useful in Neural Machine Translation?cited by 45
  15. 2019A comparison of end-to-end models for long-form speech recognitioncited by 42
  16. 2023Sparks of Large Audio Models: A Survey and Outlookcited by 41
  17. 2019A Comparative Study on End-to-end Speech to Text Translationcited by 40
  18. 2021RealTranS: End-to-End Simultaneous Speech Translation with Convolutional Weighted-Shrinking Transformercited by 40
  19. 2022Information-Transport-based Policy for Simultaneous Translationcited by 40
  20. 2024Navigating the Metrics Maze: Reconciling Score Magnitudes and Accuraciescited by 40
  21. 2018Low-Latency Neural Speech Translationcited by 37
  22. 2020Streaming Transformer-based Acoustic Models Using Self-attention with Augmented Memorycited by 35
  23. 2023AlignAtt: Using Attention-based Audio-Translation Alignments as a Guide for Simultaneous Speech Translationcited by 35
  24. 2022Attention as a Guide for Simultaneous Speech Translationcited by 32
  25. 2020Re-translation versus Streaming for Simultaneous Translationcited by 31
  26. 2021Direct Simultaneous Speech-to-Text Translation Assisted by Synchronized Streaming ASRcited by 31
  27. 2022Large-Scale Streaming End-to-End Speech Translation with Neural Transducerscited by 31
  28. 2020Streaming Simultaneous Speech Translation with Augmented Memory Transformercited by 30
  29. 2021Learning When to Translate for Streaming Speechcited by 30
  30. 2022Noise-robust Speech Recognition with 10 Minutes Unparalleled In-domain Datacited by 30
  31. 2022CUNI-KIT System for Simultaneous Speech Translation Task at IWSLT 2022cited by 30
  32. 2019Thinking Slow about Latency Evaluation for Simultaneous Machine Translationcited by 28
  33. 2022SHAS: Approaching optimal Segmentation for End-to-End Speech Translationcited by 26
  34. 2022End-to-End Speech Translation for Code Switched Speechcited by 26
  35. 2019Re-Translation Strategies For Long Form, Simultaneous, Spoken Language Translationcited by 25
  36. 2024Speech Translation with Speech Foundation Models and Large Language Models: What is There and What is Missing?cited by 23
  37. 2022Does Simultaneous Speech Translation need Simultaneous Models?cited by 22
  38. 2023End-to-End Simultaneous Speech Translation with Differentiable Segmentationcited by 22

50 most cited of 120 All 120 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 10; 2026: 7.

17 papers.

Sort:

  1. 2025Seed LiveInterpret 2.0: End-to-end Simultaneous Speech-to-speech Translation with Your Voicecited by 9
  2. 2025InfiniSST: Simultaneous Translation of Unbounded Speech with Large Language Modelcited by 8
  3. 2025Simultaneous Translation with Offline Speech and LLM Models in CUNI Submission to IWSLT 2025cited by 8
  4. 2025Hearing to Translate: The Effectiveness of Speech Modality Integration into LLMscited by 7
  5. 2025Prepending or Cross-Attention for Speech-to-Text? An Empirical Comparisoncited by 5
  6. 2025Better Late Than Never: Meta-Evaluation of Latency Metrics for Simultaneous Speech-to-Text Translationcited by 4
  7. 2025Simulstream: Open-Source Toolkit for Evaluation and Demonstration of Streaming Speech-to-Text Translation Systemscited by 4
  8. 2025CMU's IWSLT 2025 Simultaneous Speech Translation Systemcited by 2
  9. 2026DOA: Training-Free Decoder-Only Attention Policy for Long-Form Simultaneous Translation with SpeechLLMscited by 1
  10. 2026Omni-Streaming Thinkingcited by 1
  11. 2025Toward Machine Interpreting: Lessons from Human Interpreting Studiescited by 0
  12. 2025OmniFusion: Simultaneous Multilingual Multimodal Translations via Modular Fusioncited by 0
5 more
  1. 2026What do the metrics mean? A critical analysis of the use of Automated Evaluation Metrics in Interpretingcited by 0
  2. 2026SimulU: Training-free Policy for Long-form Simultaneous Speech-to-Speech Translationcited by 0
  3. 2026A Pocket Offline Model for Simultaneous Speech Translation as CUNI Submission to IWSLT 2026cited by 0
  4. 2026Bridging the Usability Gap: Lessons from Interpreting Studies for Machine Interpreting Designcited by 0
  5. 2026The Role of Disfluencies in Speech Translationcited by 0