mlsanityData to 7 Oct 2026

StreamSpeech: Simultaneous Speech-to-Speech Translation with Multi-task Learning

arXiv 2406.03049 ↗First posted 5 Jun 2024cs.CL cs.AI cs.SDcited by 20

Not on the map. Why?

Abstract (arXiv)

Simultaneous speech-to-speech translation (Simul-S2ST, a.k.a streaming speech translation) outputs target speech while receiving streaming speech inputs, which is critical for real-time communication. Beyond accomplishing translation between speech, Simul-S2ST requires a policy to control the model to generate corresponding target speech at the opportune moment within speech inputs, thereby posing a double challenge of translation and policy. In this paper, we propose StreamSpeech, a direct Simul-S2ST model that jointly learns translation and simultaneous policy in a unified framework of multi-task learning. Adhering to a multi-task learning approach, StreamSpeech can perform offline and simultaneous speech recognition, speech translation and speech synthesis via an "All-in-One" seamless model. Experiments on CVSS benchmark demonstrate that StreamSpeech achieves state-of-the-art performance in both offline S2ST and Simul-S2ST tasks. Besides, StreamSpeech is able to present high-quality intermediate results (i.e., ASR or translation results) during simultaneous translation process, offering a more comprehensive real-time communication experience.

Key facts. Cited by 20 papers in the map (data through 7 Oct 2026). Builds on 50 of its 67 references that are in the map. First posted on arXiv 5 Jun 2024.

Builds on

50 of 67 references.

References · 50

  1. 2017Attention Is All You Needcited by 59,932
  2. 2022Robust Speech Recognition via Large-Scale Weak Supervisioncited by 3,972
  3. 2020wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representationscited by 3,853
  4. 2018SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processingcited by 1,973
  5. 2020Conformer: Convolution-augmented Transformer for Speech Recognitioncited by 1,838
  6. 2018A Call for Clarity in Reporting BLEU Scorescited by 1,790
  7. 2019fairseq: A Fast, Extensible Toolkit for Sequence Modelingcited by 1,486
  8. 2020HiFi-GAN: Generative Adversarial Networks for Efficient and High Fidelity Speech Synthesiscited by 1,139
  9. 2017Non-Autoregressive Neural Machine Translationcited by 457
  10. 2021Direct speech-to-speech translation with discrete unitscited by 148
  11. 2019Direct speech-to-speech translation with a sequence-to-sequence modelcited by 118
  12. 2019Monotonic Infinite Lookback Attention for Simultaneous Machine Translationcited by 103
38 more
  1. 2016Learning to Translate in Real-time with Neural Machine Translationcited by 97
  2. 2020Non-Autoregressive Machine Translation with Latent Alignmentscited by 96
  3. 2020SimulEval: An Evaluation Toolkit for Simultaneous Translationcited by 94
  4. 2019Monotonic Multihead Attentioncited by 91
  5. 2016Can neural machine translation do simultaneous translation?cited by 85
  6. 2020SimulMT to SimulST: Adapting Simultaneous Text Translation to End-to-End Simultaneous Speech Translationcited by 80
  7. 2020CoVoST 2 and Massively Multilingual Speech-to-Text Translationcited by 73
  8. 2021Translatotron 2: High-quality direct speech-to-speech translation with voice preservationcited by 69
  9. 2022UnitY: Two-pass Direct Speech-to-speech Translation with Discrete Unitscited by 60
  10. 2020Efficient Wait-k Models for Simultaneous Machine Translationcited by 59
  11. 2022Over-Generation Cannot Be Rewarded: Length-Adaptive Average Lagging for Simultaneous Speech Translationcited by 56
  12. 2021Improving Speech Translation by Understanding and Learning from the Auxiliary Text Translation Taskcited by 54
  13. 2020A General Multi-Task Learning Framework to Leverage Text Data for Speech to Text Taskscited by 51
  14. 2020Simultaneous Translation Policies: From Fixed to Adaptivecited by 42
  15. 2022Information-Transport-based Policy for Simultaneous Translationcited by 40
  16. 2021RealTranS: End-to-End Simultaneous Speech Translation with Convolutional Weighted-Shrinking Transformercited by 39
  17. 2021Universal Simultaneous Machine Translation with Mixture-of-Experts Wait-k Policycited by 32
  18. 2022Attention as a Guide for Simultaneous Speech Translationcited by 32
  19. 2021Direct Simultaneous Speech-to-Text Translation Assisted by Synchronized Streaming ASRcited by 31
  20. 2020Future-Guided Incremental Transformer for Simultaneous Translationcited by 30
  21. 2021Learning When to Translate for Streaming Speechcited by 30
  22. 2023Hidden Markov Transformer for Simultaneous Machine Translationcited by 24
  23. 2023End-to-End Simultaneous Speech Translation with Differentiable Segmentationcited by 22
  24. 2022Modeling Dual Read/Write Paths for Simultaneous Machine Translationcited by 21
  25. 2022Gaussian Multi-head Attention for Simultaneous Machine Translationcited by 17
  26. 2022Reducing Position Bias in Simultaneous Machine Translation with Length-Aware Frameworkcited by 16
  27. 2022Wait-info Policy: Balancing Source and Target at Information Level for Simultaneous Machine Translationcited by 16
  28. 2022Average Token Delay: A Latency Metric for Simultaneous Translationcited by 15
  29. 2023Learning Optimal Policy for Simultaneous Machine Translation via Binary Searchcited by 15
  30. 2022Turning Fixed to Adaptive: Integrating Post-Evaluation into Simultaneous Machine Translationcited by 14
  31. 2024SiLLM: Large Language Models for Simultaneous Machine Translationcited by 13
  32. 2023Non-autoregressive Streaming Transformer for Simultaneous Translationcited by 12
  33. 2023Glancing Future for Simultaneous Machine Translationcited by 10
  34. 2023DASpeech: Directed Acyclic Transformer for Fast and High-quality Speech-to-Speech Translationcited by 9
  35. 2023Simultaneous Machine Translation with Tailored Referencecited by 9
  36. 2023Adapting Offline Speech Translation Models for Streaming with Future-Aware Distillation and Inferencecited by 6
  37. 2023Unified Segment-to-Segment Framework for Simultaneous Sequence Generationcited by 6
  38. 2023Learning When to Speak: Latency and Quality Trade-offs for Simultaneous Speech-to-Speech Translation with Offline Modelscited by 2

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 5; 2025: 7; 2026: 8.

20 papers.

Sort:

By area:

  1. 2024Lyra: An Efficient and Speech-Centric Framework for Omni-Cognitioncited by 20
  2. 2024How "Real" is Your Real-Time Simultaneous Speech-to-Text Translation System?cited by 17
  3. 2025High-Fidelity Simultaneous Speech-To-Speech Translationcited by 15
  4. 2024Towards Achieving Human Parity on End-to-end Simultaneous Speech Translation via LLM Agentcited by 14
  5. 2025OpenOmni: Advancing Open-Source Omnimodal Large Language Models with Progressive Multimodal Alignment and Real-Time Self-Aware Emotional Speech Synthesiscited by 10
  6. 2025Seed LiveInterpret 2.0: End-to-end Simultaneous Speech-to-speech Translation with Your Voicecited by 9
  7. 2026Simultaneous Speech-to-Speech Translation Without Aligned Datacited by 7
  8. 2024Overcoming Non-monotonicity in Transducer-based Streaming Generationcited by 4
  9. 2024Transcribing and Translating, Fast and Slow: Joint Speech Translation and Recognitioncited by 2
  10. 2025SimulMEGA: MoE Routers are Advanced Policy Makers for Simultaneous Speech Translationcited by 2
  11. 2026Multi-Stream LLMs: Unblocking Language Models with Parallel Streams of Thoughts, Inputs and Outputscited by 2
  12. 2025MTP-S2UT: Enhancing Speech-to-Speech Translation Quality with Multi-token Predictioncited by 1
8 more
  1. 2025Overcoming Latency Bottlenecks in On-Device Speech Translation: A Cascaded Approach with Alignment-Based Streaming MTcited by 0
  2. 2025Asynchronous Pipeline Parallelism for Real-Time Multilingual Lip Synchronization in Video Communication Systemscited by 0
  3. 2026SimulU: Training-free Policy for Long-form Simultaneous Speech-to-Speech Translationcited by 0
  4. 2026Streaming Speech-to-Text Translation with a SpeechLLMcited by 0
  5. 2026Multilingual Multi-Speaker Unit Vocoders: A Systematic Analysis of Discrete Speech Representationscited by 0
  6. 2026NaturalFlow: Reducing Disruptive Pauses for Natural Speech Flow in Simultaneous Speech-to-Speech Translationcited by 0
  7. 2026EviSI: An Evidence-Based Evaluation Agent for Simultaneous Interpretingcited by 0
  8. 2026Attention-Based Adaptive Policies for Simultaneous Speech-to-Text Translationcited by 0