mlsanityData to 14 Oct 2026

Recent Advances in End-to-End Simultaneous Speech Translation

arXiv 2406.00497 ↗First posted 1 Jun 2024cs.SD cs.AI cs.CLcited by 5

Not on the map. Why?

Abstract (arXiv)

Simultaneous speech translation (SimulST) is a demanding task that involves generating translations in real-time while continuously processing speech input. This paper offers a comprehensive overview of the recent developments in SimulST research, focusing on four major challenges. Firstly, the complexities associated with processing lengthy and continuous speech streams pose significant hurdles. Secondly, satisfying real-time requirements presents inherent difficulties due to the need for immediate translation output. Thirdly, striking a balance between translation quality and latency constraints remains a critical challenge. Finally, the scarcity of annotated data adds another layer of complexity to the task. Through our exploration of these challenges and the proposed solutions, we aim to provide valuable insights into the current landscape of SimulST research and suggest promising directions for future exploration.

Key facts. Cited by 5 papers in the map (data through 14 Oct 2026). Builds on 38 of its 49 references that are in the map. First posted on arXiv 1 Jun 2024.

Builds on

38 of 49 references.

References · 38

  1. 2017Attention Is All You Needcited by 60,361
  2. 2022Robust Speech Recognition via Large-Scale Weak Supervisioncited by 4,057
  3. 2023A Survey of Large Language Modelscited by 2,576
  4. 2018A Call for Clarity in Reporting BLEU Scorescited by 1,799
  5. 2023AudioPaLM: A Large Language Model That Can Speak and Listencited by 343
  6. 2023Seamless: Multilingual Expressive and Streaming Speech Translationcited by 192
  7. 2020SimulEval: An Evaluation Toolkit for Simultaneous Translationcited by 94
  8. 2020SimulMT to SimulST: Adapting Simultaneous Text Translation to End-to-End Simultaneous Speech Translationcited by 81
  9. 2020Unbabel's Participation in the WMT20 Metrics Shared Taskcited by 56
  10. 2020Low-Latency Sequence-to-Sequence Speech Recognition and Translation by Partial Hypothesis Selectioncited by 49
  11. 2021RealTranS: End-to-End Simultaneous Speech Translation with Convolutional Weighted-Shrinking Transformercited by 40
  12. 2018Low-Latency Neural Speech Translationcited by 37
26 more
  1. 2023AlignAtt: Using Attention-based Audio-Translation Alignments as a Guide for Simultaneous Speech Translationcited by 35
  2. 2022Attention as a Guide for Simultaneous Speech Translationcited by 32
  3. 2021Direct Simultaneous Speech-to-Text Translation Assisted by Synchronized Streaming ASRcited by 31
  4. 2020Streaming Simultaneous Speech Translation with Augmented Memory Transformercited by 30
  5. 2021Learning When to Translate for Streaming Speechcited by 30
  6. 2019Thinking Slow about Latency Evaluation for Simultaneous Machine Translationcited by 28
  7. 2022Does Simultaneous Speech Translation need Simultaneous Models?cited by 22
  8. 2023End-to-End Simultaneous Speech Translation with Differentiable Segmentationcited by 22
  9. 2023Hybrid Transducer and Attention based Encoder-Decoder Modeling for Speech-to-Text Taskscited by 20
  10. 2022Average Token Delay: A Latency Metric for Simultaneous Translationcited by 15
  11. 2023Recent Advances in Direct Speech-to-text Translationcited by 15
  12. 2021An Empirical Study of End-to-end Simultaneous Speech Translation Decoding Strategiescited by 14
  13. 2022Exploring Continuous Integrate-and-Fire for Adaptive Simultaneous Speech Translationcited by 14
  14. 2022Blockwise Streaming Transformer for Spoken Language Understanding and Simultaneous Speech Translationcited by 11
  15. 2022MT Metrics Correlate with Human Ratings of Simultaneous Speech Translationcited by 9
  16. 2023ComSL: A Composite Speech-Language Model for End-to-End Speech-to-Text Translationcited by 9
  17. 2023End-to-End Speech-to-Text Translation: A Surveycited by 9
  18. 2023Incremental Blockwise Beam Search for Simultaneous Speech Translation with Controllable Quality-Latency Tradeoffcited by 8
  19. 2021Decision Attentive Regularization to Improve Simultaneous Speech Translation Systemscited by 6
  20. 2023Tagged End-to-End Simultaneous Speech Translation Training using Simultaneous Interpretation Datacited by 6
  21. 2021Impact of Encoding and Segmentation Strategies on End-to-End Simultaneous Speech Translationcited by 4
  22. 2023Shiftable Context: Addressing Training-Inference Context Mismatch in Simultaneous Speech Translationcited by 4
  23. 2023Implicit Memory Transformer for Computationally Efficient Simultaneous Speech Translationcited by 4
  24. 2023Long-form Simultaneous Speech Translation: Thesis Proposalcited by 3
  25. 2022Multilingual Simultaneous Speech Translationcited by 2
  26. 2023Improving Stability in Simultaneous Speech Translation: A Revision-Controllable Decoding Approachcited by 2

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 2; 2025: 2; 2026: 1.

5 papers.

  1. 2024How "Real" is Your Real-Time Simultaneous Speech-to-Text Translation System?cited by 17
  2. 2025FLEXI: Benchmarking Full-duplex Human-LLM Speech Interactioncited by 15
  3. 2024A Modular-based Strategy for Mitigating Gradient Conflicts in Simultaneous Speech Translationcited by 1
  4. 2025Direct Simultaneous Translation Activation for Large Audio-Language Modelscited by 0
  5. 2026Attention-Based Adaptive Policies for Simultaneous Speech-to-Text Translationcited by 0