mlsanityData to 14 Oct 2026

SimulU: Training-free Policy for Long-form Simultaneous Speech-to-Speech Translation

arXiv 2603.16924 ↗First posted 11 Mar 2026eess.AS cs.AI cs.CLcited by 0

Not on the map. Why?

Abstract (arXiv)

Simultaneous speech-to-speech translation (SimulS2S) is essential for real-time multilingual communication, with increasing integration into meeting and streaming platforms. Despite this, SimulS2S remains underexplored in research, where current solutions often rely on resource-intensive training procedures and operate on short-form, pre-segmented utterances, failing to generalize to continuous speech. To bridge this gap, we propose SimulU, the first training-free policy for long-form SimulS2S. SimulU adopts history management and speech output selection strategies that exploit cross-attention in pre-trained end-to-end models to regulate both input history and output generation. Evaluations on MuST-C across 8 languages show that SimulU achieves a better or comparable quality-latency trade-off against strong cascaded models. By eliminating the need for ad-hoc training, SimulU offers a promising path to end-to-end SimulS2S in realistic, long-form scenarios.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 29 of its 41 references that are in the map. First posted on arXiv 11 Mar 2026.

Builds on

29 of 41 references.

References · 29

  1. 2024The Llama 3 Herd of Modelscited by 14,715
  2. 2022Robust Speech Recognition via Large-Scale Weak Supervisioncited by 4,057
  3. 2022No Language Left Behind: Scaling Human-Centered Machine Translationcited by 1,312
  4. 2020HiFi-GAN: Generative Adversarial Networks for Efficient and High Fidelity Speech Synthesiscited by 1,154
  5. 2021XLS-R: Self-supervised Cross-lingual Speech Representation Learning at Scalecited by 601
  6. 2022UTMOS: UTokyo-SaruLab System for VoiceMOS Challenge 2022cited by 390
  7. 2021W2v-BERT: Combining Contrastive Learning and Masked Language Modeling for Self-Supervised Speech Pre-Trainingcited by 347
  8. 2023SeamlessM4T: Massively Multilingual & Multimodal Machine Translationcited by 125
  9. 2019Direct speech-to-speech translation with a sequence-to-sequence modelcited by 118
  10. 2023SONAR: Sentence-Level Multimodal and Language-Agnostic Representationscited by 108
  11. 2020SimulEval: An Evaluation Toolkit for Simultaneous Translationcited by 94
  12. 2020Speech Translation and the End-to-End Promise: Taking Stock of Where We Arecited by 77
17 more
  1. 2022UnitY: Two-pass Direct Speech-to-speech Translation with Discrete Unitscited by 60
  2. 2025Canary-1B-v2 & Parakeet-TDT-0.6B-v3: Efficient and High-Performance Models for Multilingual ASR and ASTcited by 60
  3. 2020Low-Latency Sequence-to-Sequence Speech Recognition and Translation by Partial Hypothesis Selectioncited by 49
  4. 2024Fish-Speech: Leveraging Large Language Models for Advanced Multilingual Text-to-Speech Synthesiscited by 48
  5. 2023AlignAtt: Using Attention-based Audio-Translation Alignments as a Guide for Simultaneous Speech Translationcited by 35
  6. 2022CUNI-KIT System for Simultaneous Speech Translation Task at IWSLT 2022cited by 30
  7. 2024StreamSpeech: Simultaneous Speech-to-Speech Translation with Multi-task Learningcited by 21
  8. 2024StreamAtt: Direct Streaming Speech-to-Text Translation with Attention-based Audio History Selectioncited by 19
  9. 2024How "Real" is Your Real-Time Simultaneous Speech-to-Text Translation System?cited by 17
  10. 2025High-Fidelity Simultaneous Speech-To-Speech Translationcited by 15
  11. 2024Speech is More Than Words: Do Speech-to-Text Translation Systems Leverage Prosody?cited by 13
  12. 2025Seed LiveInterpret 2.0: End-to-end Simultaneous Speech-to-speech Translation with Your Voicecited by 9
  13. 2025InfiniSST: Simultaneous Translation of Unbounded Speech with Large Language Modelcited by 8
  14. 2025Simultaneous Translation with Offline Speech and LLM Models in CUNI Submission to IWSLT 2025cited by 8
  15. 2026Simultaneous Speech-to-Speech Translation Without Aligned Datacited by 7
  16. 2024Textless Streaming Speech-to-Speech Translation using Semantic Speech Tokenscited by 5
  17. 2025SimulS2S-LLM: Unlocking Simultaneous Inference of Speech LLMs for Speech-to-Speech Translationcited by 3

Built on by

Not cited yet.