mlsanityData to 7 Oct 2026

Long-form Simultaneous Speech Translation: Thesis Proposal

arXiv 2310.11141 ↗First posted 17 Oct 2023cs.CL cs.AI cs.SDcited by 3

Not on the map. Why?

Abstract (arXiv)

Simultaneous speech translation (SST) aims to provide real-time translation of spoken language, even before the speaker finishes their sentence. Traditionally, SST has been addressed primarily by cascaded systems that decompose the task into subtasks, including speech recognition, segmentation, and machine translation. However, the advent of deep learning has sparked significant interest in end-to-end (E2E) systems. Nevertheless, a major limitation of most approaches to E2E SST reported in the current literature is that they assume that the source speech is pre-segmented into sentences, which is a significant obstacle for practical, real-world applications. This thesis proposal addresses end-to-end simultaneous speech translation, particularly in the long-form setting, i.e., without pre-segmentation. We present a survey of the latest advancements in E2E SST, assess the primary obstacles in SST and its relevance to long-form scenarios, and suggest approaches to tackle these challenges.

Key facts. Cited by 3 papers in the map (data through 7 Oct 2026). Builds on 42 of its 81 references that are in the map. First posted on arXiv 17 Oct 2023.

Builds on

42 of 81 references.

References · 42

  1. 2022Robust Speech Recognition via Large-Scale Weak Supervisioncited by 3,972
  2. 2019Transformer-XL: Attentive Language Models Beyond a Fixed-Length Contextcited by 1,830
  3. 2018A Call for Clarity in Reporting BLEU Scorescited by 1,790
  4. 2023Google USM: Scaling Automatic Speech Recognition Beyond 100 Languagescited by 228
  5. 2023Fast Conformer with Linearly Scalable Attention for Efficient Speech Recognitioncited by 120
  6. 2019Europarl-ST: A Multilingual Corpus For Speech Translation Of Parliamentary Debatescited by 114
  7. 2021The Multilingual TEDx Corpus for Speech Recognition and Translationcited by 101
  8. 2017Neural Machine Translation with Extended Contextcited by 95
  9. 2020SimulEval: An Evaluation Toolkit for Simultaneous Translationcited by 94
  10. 2016Can neural machine translation do simultaneous translation?cited by 85
  11. 2019Recognizing long-form speech using streaming end-to-end modelscited by 77
  12. 2020Speech Translation and the End-to-End Promise: Taking Stock of Where We Arecited by 77
30 more
  1. 2020Unbabel's Participation in the WMT20 Metrics Shared Taskcited by 56
  2. 2022Over-Generation Cannot Be Rewarded: Length-Adaptive Average Lagging for Simultaneous Speech Translationcited by 56
  3. 2021Cascade versus Direct Speech Translation: Do the Differences Still Make a Difference?cited by 54
  4. 2018Incremental Decoding and Training Methods for Simultaneous Translation in Neural Machine Translationcited by 51
  5. 2021Measuring and Increasing Context Usage in Context-Aware Machine Translationcited by 48
  6. 2019When and Why is Document-level Context Useful in Neural Machine Translation?cited by 45
  7. 2019A comparison of end-to-end models for long-form speech recognitioncited by 42
  8. 2018Low-Latency Neural Speech Translationcited by 35
  9. 2020Re-translation versus Streaming for Simultaneous Translationcited by 30
  10. 2020Streaming Simultaneous Speech Translation with Augmented Memory Transformercited by 30
  11. 2022CUNI-KIT System for Simultaneous Speech Translation Task at IWSLT 2022cited by 30
  12. 2019Thinking Slow about Latency Evaluation for Simultaneous Machine Translationcited by 28
  13. 2020RNN-T Models Fail to Generalize to Out-of-Domain Audio: Causes and Solutionscited by 26
  14. 2022SHAS: Approaching optimal Segmentation for End-to-End Speech Translationcited by 26
  15. 2023Turning Whisper into Real-Time Transcription Systemcited by 23
  16. 2020End-to-End Automatic Speech Recognition Integrated With CTC-Based Voice Activity Detectioncited by 20
  17. 2021Beyond Voice Activity Detection: Hybrid Audio Segmentation for Direct Speech Translationcited by 17
  18. 2022E2E Segmenter: Joint Segmenting and Decoding for Long-Form ASRcited by 17
  19. 2021End-to-End Speech Translation with Pre-trained Models and Adapters: UPC at IWSLT 2021cited by 16
  20. 2022Learn To Remember: Transformer with Recurrent Memory for Document-Level Machine Translationcited by 16
  21. 2021Diverse Pretrained Context Encodings Improve Document Translationcited by 13
  22. 2020Contextualized Translation of Automatically Segmented Speechcited by 12
  23. 2021Stream-level Latency Evaluation for Simultaneous Machine Translationcited by 12
  24. 2021Input Length Matters: Improving RNN-T and MWER Training for Long-form Telephony Speech Recognitioncited by 10
  25. 2023ESPnet-ST-v2: Multipurpose Spoken Language Translation Toolkitcited by 10
  26. 2022From Simultaneous to Streaming Machine Translation by Leveraging Streaming Historycited by 9
  27. 2022Don't Discard Fixed-Window Audio Segmentation in Speech-to-Text Translationcited by 8
  28. 2023Incremental Blockwise Beam Search for Simultaneous Speech Translation with Controllable Quality-Latency Tradeoffcited by 8
  29. 2022Speech Segmentation Optimization using Segmented Bilingual Speech Corpus for End-to-end Speech Translationcited by 7
  30. 2023Long-Form End-to-End Speech Translation via Latent Alignment Segmentationcited by 6

Built on by

Cited by year

Citing papers per year: 2024: 3.

3 papers.

  1. 2024How "Real" is Your Real-Time Simultaneous Speech-to-Text Translation System?cited by 17
  2. 2024Recent Advances in End-to-End Simultaneous Speech Translationcited by 5
  3. 2024CA*: Addressing Evaluation Pitfalls in Computation-Aware Latency for Simultaneous Speech Translationcited by 0