SimulTron: On-Device Simultaneous Speech to Speech Translation
Not on the map. Why?
Abstract (arXiv)
Simultaneous speech-to-speech translation (S2ST) holds the promise of breaking down communication barriers and enabling fluid conversations across languages. However, achieving accurate, real-time translation through mobile devices remains a major challenge. We introduce SimulTron, a novel S2ST architecture designed to tackle this task. SimulTron is a lightweight direct S2ST model that uses the strengths of the Translatotron framework while incorporating key modifications for streaming operation, and an adjustable fixed delay. Our experiments show that SimulTron surpasses Translatotron 2 in offline evaluations. Furthermore, real-time evaluations reveal that SimulTron improves upon the performance achieved by Translatotron 1. Additionally, SimulTron achieves superior BLEU scores and latency compared to previous real-time S2ST method on the MuST-C dataset. Significantly, we have successfully deployed SimulTron on a Pixel 7 Pro device, show its potential for simultaneous S2ST on-device.
Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Builds on 28 of its 35 references that are in the map. First posted on arXiv 4 Jun 2024.
Builds on
28 of 35 references.
References · 28
- 2014Adam: A Method for Stochastic Optimizationcited by 36,960
- 2020wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representationscited by 3,853
- 2021HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Unitscited by 2,301
- 2021SoundStream: An End-to-End Neural Audio Codeccited by 756
- 2022High Fidelity Neural Audio Compressioncited by 673
- 2021W2v-BERT: Combining Contrastive Learning and Masked Language Modeling for Self-Supervised Speech Pre-Trainingcited by 340
- 2023AudioPaLM: A Large Language Model That Can Speak and Listencited by 333
- 2023Seamless: Multilingual Expressive and Streaming Speech Translationcited by 190
- 2021Direct speech-to-speech translation with discrete unitscited by 148
- 2019Direct speech-to-speech translation with a sequence-to-sequence modelcited by 118
- 2019Lingvo: a Modular and Scalable Framework for Sequence-to-Sequence Modelingcited by 86
- 2021Translatotron 2: High-quality direct speech-to-speech translation with voice preservationcited by 69
16 more
- 2022UnitY: Two-pass Direct Speech-to-speech Translation with Discrete Unitscited by 60
- 2018Leveraging Weakly Supervised Data to Improve End-to-End Speech-to-Text Translationcited by 57
- 2020UWSpeech: Speech to Speech Translation for Unwritten Languagescited by 43
- 2022TranSpeech: Speech-to-Speech Translation With Bilateral Perturbationcited by 34
- 2019Speech-to-speech Translation between Untranscribed Unknown Languagescited by 27
- 2022Textless Direct Speech-to-Speech Translation with Discrete Speech Representationcited by 19
- 2020Simultaneous Speech-to-Speech Translation System with Neural Incremental ASR, MT, and TTScited by 12
- 2022Simple and Effective Unsupervised Speech Translationcited by 12
- 2023Translatotron 3: Speech to Speech Translation with Monolingual Datacited by 12
- 2021Direct Simultaneous Speech-to-Speech Translation with Variational Monotonic Multihead Attentioncited by 9
- 2022Joint Pre-Training with Speech and Bilingual Text for Direct Speech to Speech Translationcited by 9
- 2023Improving Cascaded Unsupervised Speech Translation with Denoising Back-translationcited by 5
- 2021From Start to Finish: Latency Reduction Strategies for Incremental Speech Synthesis in Simultaneous Speech-to-Speech Translationcited by 3
- 2022Real time spectrogram inversion on mobile phonecited by 2
- 2023Learning When to Speak: Latency and Quality Trade-offs for Simultaneous Speech-to-Speech Translation with Offline Modelscited by 2
- 2022Streaming Parrotron for on-device speech-to-speech conversioncited by 1
Built on by
Citing papers per year: 2024: 1.
1 paper.
- 2024Direct Speech-to-Speech Neural Machine Translation: A Survey13 Nov 2024