mlsanityData to 7 Oct 2026

Learning When to Translate for Streaming Speech

arXiv 2109.07368 ↗First posted 15 Sep 2021cs.CL eess.AScited by 30

Not on the map. Why?

Abstract (arXiv)

How to find proper moments to generate partial sentence translation given a streaming speech input? Existing approaches waiting-and-translating for a fixed duration often break the acoustic units in speech, since the boundaries between acoustic units in speech are not even. In this paper, we propose MoSST, a simple yet effective method for translating streaming speech content. Given a usually long speech sequence, we develop an efficient monotonic segmentation module inside an encoder-decoder model to accumulate acoustic information incrementally and detect proper speech unit boundaries for the input in speech translation task. Experiments on multiple translation directions of the MuST-C dataset show that MoSST outperforms existing methods and achieves the best trade-off between translation quality (BLEU) and latency. Our code is available at https://github.com/dqqcasia/mosst.

Key facts. Cited by 30 papers in the map (data through 7 Oct 2026). Builds on 47 of its 65 references that are in the map. First posted on arXiv 15 Sep 2021.

Builds on

47 of 65 references.

References · 47

  1. 2017Attention Is All You Needcited by 59,932
  2. 2020wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representationscited by 3,853
  3. 2018SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processingcited by 1,973
  4. 2019fairseq: A Fast, Extensible Toolkit for Sequence Modelingcited by 1,486
  5. 2020fairseq S2T: Fast Speech-to-Text Modeling with fairseqcited by 174
  6. 2017Sequence-to-Sequence Models Can Directly Translate Foreign Speechcited by 121
  7. 2016Listen and Translate: A Proof of Concept for End-to-End Speech-to-Text Translationcited by 113
  8. 2019Monotonic Infinite Lookback Attention for Simultaneous Machine Translationcited by 103
  9. 2016Learning to Translate in Real-time with Neural Machine Translationcited by 97
  10. 2020SimulEval: An Evaluation Toolkit for Simultaneous Translationcited by 94
  11. 2019Monotonic Multihead Attentioncited by 91
  12. 2016Can neural machine translation do simultaneous translation?cited by 85
35 more
  1. 2020ESPnet-ST: All-in-One Speech Translation Toolkitcited by 82
  2. 2020SimulMT to SimulST: Adapting Simultaneous Text Translation to End-to-End Simultaneous Speech Translationcited by 80
  3. 2019CIF: Continuous Integrate-and-Fire for End-to-End Speech Recognitioncited by 76
  4. 2018Pre-training on high-resource speech recognition improves low-resource speech-to-text translationcited by 70
  5. 2018End-to-End Automatic Speech Translation of Audiobookscited by 65
  6. 2020Efficient Wait-k Models for Simultaneous Machine Translationcited by 59
  7. 2019End-to-End Speech Translation with Knowledge Distillationcited by 58
  8. 2018Leveraging Weakly Supervised Data to Improve End-to-End Speech-to-Text Translationcited by 57
  9. 2020Curriculum Pre-training for End-to-End Speech Translationcited by 57
  10. 2020Bridging the Modality Gap for Speech-to-Text Translationcited by 55
  11. 2020Dual-decoder Transformer for Joint Automatic Speech Recognition and Multilingual Speech Translationcited by 54
  12. 2021Lightweight Adapter Tuning for Multilingual Speech Translationcited by 53
  13. 2018Incremental Decoding and Training Methods for Simultaneous Translation in Neural Machine Translationcited by 51
  14. 2021Learning Shared Semantic Space for Speech-to-Text Translationcited by 49
  15. 2021End-to-end Speech Translation via Cross-modal Progressive Trainingcited by 47
  16. 2021RealTranS: End-to-End Simultaneous Speech Translation with Convolutional Weighted-Shrinking Transformercited by 39
  17. 2019Simpler and Faster Learning of Adaptive Policies for Simultaneous Translationcited by 36
  18. 2021Direct Simultaneous Speech-to-Text Translation Assisted by Synchronized Streaming ASRcited by 31
  19. 2020Re-translation versus Streaming for Simultaneous Translationcited by 30
  20. 2020Streaming Simultaneous Speech Translation with Augmented Memory Transformercited by 30
  21. 2019Thinking Slow about Latency Evaluation for Simultaneous Machine Translationcited by 28
  22. 2020Consecutive Decoding for Speech-to-text Translationcited by 28
  23. 2019Analyzing ASR pretraining for low-resource speech-to-text translationcited by 27
  24. 2020Adaptive Feature Selection for End-to-End Speech Translationcited by 25
  25. 2019Re-Translation Strategies For Long Form, Simultaneous, Spoken Language Translationcited by 24
  26. 2020NeurST: Neural Speech Translation Toolkitcited by 23
  27. 2019Synchronous Speech Recognition and Speech-to-Text Translation with Interactive Decodingcited by 21
  28. 2019On Using SpecAugment for End-to-End Speech Translationcited by 19
  29. 2021Efficiently Fusing Pretrained Acoustic and Linguistic Encoders for Low-resource Speech Recognitioncited by 17
  30. 2020SkinAugment: Auto-Encoding Speaker Conversions for Automatic Speech Translationcited by 13
  31. 2020MAM: Masked Acoustic Modeling for End-to-End Speech-to-Text Translationcited by 13
  32. 2018Structured-based Curriculum Learning for End-to-end English-Japanese Speech Translationcited by 11
  33. 2020On Sparsifying Encoder Outputs in Sequence-to-Sequence Modelscited by 8
  34. 2020Opportunistic Decoding with Timely Correction for Simultaneous Translationcited by 8
  35. 2018STACL: Simultaneous Translation with Implicit Anticipation and Controllable Latency using Prefix-to-Prefix Frameworkcited by 4

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 4; 2023: 6; 2024: 9; 2025: 10; 2026: 1.

30 papers.

Sort:

By area:

  1. 2023SciMON: Scientific Inspiration Machines Optimized for Noveltycited by 174
  2. 2022Information-Transport-based Policy for Simultaneous Translationcited by 40
  3. 2023Turning Whisper into Real-Time Transcription Systemcited by 23
  4. 2023End-to-End Simultaneous Speech Translation with Differentiable Segmentationcited by 22
  5. 2024StreamSpeech: Simultaneous Speech-to-Speech Translation with Multi-task Learningcited by 20
  6. 2024StreamAtt: Direct Streaming Speech-to-Text Translation with Attention-based Audio History Selectioncited by 19
  7. 2024How "Real" is Your Real-Time Simultaneous Speech-to-Text Translation System?cited by 17
  8. 2022Exploring Continuous Integrate-and-Fire for Adaptive Simultaneous Speech Translationcited by 14
  9. 2024Towards Achieving Human Parity on End-to-end Simultaneous Speech Translation via LLM Agentcited by 14
  10. 2025Efficient and Adaptive Simultaneous Speech Translation with Fully Unidirectional Architecturecited by 12
  11. 2023End-to-End Speech-to-Text Translation: A Surveycited by 9
  12. 2025LLM as Effective Streaming Processor: Bridging Streaming-Batch Mismatches with Group Position Encodingcited by 9
18 more
  1. 2025InfiniSST: Simultaneous Translation of Unbounded Speech with Large Language Modelcited by 8
  2. 2024Simul-Whisper: Attention-Guided Streaming Whisper with Truncation Detectioncited by 7
  3. 2024FASST: Fast LLM-based Simultaneous Speech Translationcited by 7
  4. 2023Adapting Offline Speech Translation Models for Streaming with Future-Aware Distillation and Inferencecited by 6
  5. 2023Unified Segment-to-Segment Framework for Simultaneous Sequence Generationcited by 6
  6. 2025StreamUni: Achieving Streaming Speech Translation with a Unified Large Speech-Language Modelcited by 6
  7. 2024Recent Advances in End-to-End Simultaneous Speech Translationcited by 5
  8. 2024Label-Synchronous Neural Transducer for E2E Simultaneous Speech Translationcited by 5
  9. 2022Align, Write, Re-order: Explainable End-to-End Speech Translation via Operation Sequence Generationcited by 4
  10. 2025Large Language Models Are Read/Write Policy-Makers for Simultaneous Generationcited by 4
  11. 2022AdaTranS: Adapting with Boundary-based Shrinking for End-to-End Speech Translationcited by 3
  12. 2025SimulS2S-LLM: Unlocking Simultaneous Inference of Speech LLMs for Speech-to-Speech Translationcited by 3
  13. 2024Streaming Sequence Transduction through Dynamic Compressioncited by 2
  14. 2025SASST: Leveraging Syntax-Aware Chunking and LLMs for Simultaneous Speech Translationcited by 2
  15. 2025SimulMEGA: MoE Routers are Advanced Policy Makers for Simultaneous Speech Translationcited by 2
  16. 2025Direct Simultaneous Translation Activation for Large Audio-Language Modelscited by 0
  17. 2025SimulSense: Sense-Driven Interpreting for Efficient Simultaneous Speech Translationcited by 0
  18. 2026Dual-Stream Simultaneous Translation via 2D Grid Attentioncited by 0