# Direct Speech-to-Speech Neural Machine Translation: A Survey

arXiv: https://arxiv.org/abs/2411.14453 · HTML: https://mlsanity.com/paper/2411.14453/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- Builds on 21 of its 138 references that are in the map.
- First posted on arXiv 13 Nov 2024.

## Builds on (most cited first, 21 of 21)

- [Distilling the Knowledge in a Neural Network](https://mlsanity.com/paper/1503.02531/) (2015)
- [Representation Learning with Contrastive Predictive Coding](https://mlsanity.com/paper/1807.03748/) (2018)
- [Neural Discrete Representation Learning](https://mlsanity.com/paper/1711.00937/) (2017)
- [FastSpeech 2: Fast and High-Quality End-to-End Text to Speech](https://mlsanity.com/paper/2006.04558/) (2020)
- [Speak Foreign Languages with Your Own Voice: Cross-Lingual Neural Codec Language Modeling](https://mlsanity.com/paper/2303.03926/) (2023)
- [Can neural machine translation do simultaneous translation?](https://mlsanity.com/paper/1606.02012/) (2016)
- [Non-Attentive Tacotron: Robust and Controllable Neural TTS Synthesis Including Unsupervised Duration Modeling](https://mlsanity.com/paper/2010.04301/) (2020)
- [SpeechGen: Unlocking the Generative Power of Speech Language Models with Prompts](https://mlsanity.com/paper/2306.02207/) (2023)
- [MSLM-S2ST: A Multitask Speech Language Model for Textless Speech-to-Speech Translation with Speaker Style Preservation](https://mlsanity.com/paper/2403.12408/) (2024)
- [TransVIP: Speech to Speech Translation System with Voice and Isochrony Preservation](https://mlsanity.com/paper/2405.17809/) (2024)
- [Direct Simultaneous Speech-to-Speech Translation with Variational Monotonic Multihead Attention](https://mlsanity.com/paper/2110.08250/) (2021)
- [SeamlessExpressiveLM: Speech Language Model for Expressive Speech-to-Speech Translation with Chain-of-Thought](https://mlsanity.com/paper/2405.20410/) (2024)
- [Speech-to-Speech Translation with Discrete-Unit-Based Style Transfer](https://mlsanity.com/paper/2309.07566/) (2023)
- [Simul-LLM: A Framework for Exploring High-Quality Simultaneous Translation with Large Language Models](https://mlsanity.com/paper/2312.04691/) (2023)
- [Bitext Mining for Low-Resource Languages via Contrastive Learning](https://mlsanity.com/paper/2208.11194/) (2022)
- [Improving Cascaded Unsupervised Speech Translation with Denoising Back-translation](https://mlsanity.com/paper/2305.07455/) (2023)
- [Multilingual Speech-to-Speech Translation into Multiple Target Languages](https://mlsanity.com/paper/2307.08655/) (2023)
- [Textless Speech-to-Speech Translation With Limited Parallel Data](https://mlsanity.com/paper/2305.15405/) (2023)
- [Direct Speech-to-speech Translation without Textual Annotation using Bottleneck Features](https://mlsanity.com/paper/2212.05805/) (2022)
- [Direct Punjabi to English speech translation using discrete units](https://mlsanity.com/paper/2402.15967/) (2024)
- [SimulTron: On-Device Simultaneous Speech to Speech Translation](https://mlsanity.com/paper/2406.02133/) (2024)

## Built on by (most cited first, 6 of 6)

- [MTalk-Bench: Evaluating Speech-to-Speech Models in Multi-Turn Dialogues via Arena-style and Rubrics Protocols](https://mlsanity.com/paper/2508.18240/) (2025)
- [Locate-and-Focus: Enhancing Terminology Translation in Speech Language Models](https://mlsanity.com/paper/2507.18263/) (2025)
- [Multimodal Large Language Model-Enabled Video Translation: A Role-Oriented Survey](https://mlsanity.com/paper/2604.11283/) (2026)
- [Evaluating and Preserving Lexical Stress in English-to-Chinese Speech-to-Speech Translation](https://mlsanity.com/paper/2606.15266/) (2026)
- [Evaluating Real-Time Voice Agents: From Component Quality to Grounded Outcomes](https://mlsanity.com/paper/2609.30798/) (2026)
- [PACT: A Pragmatic Prompting Framework for Expressive and Voice-Preserving Speech-to-Speech Translation](https://mlsanity.com/paper/2610.18247/) (2026)

## Abstract (arXiv)

Speech-to-Speech Translation (S2ST) models transform speech from one language to another target language with the same linguistic information. S2ST is important for bridging the communication gap among communities and has diverse applications. In recent years, researchers have introduced direct S2ST models, which have the potential to translate speech without relying on intermediate text generation, have better decoding latency, and the ability to preserve paralinguistic and non-linguistic features. However, direct S2ST has yet to achieve quality performance for seamless communication and still lags behind the cascade models in terms of performance, especially in real-world translation. To the best of our knowledge, no comprehensive survey is available on the direct S2ST system, which beginners and advanced researchers can look upon for a quick survey. The present work provides a comprehensive review of direct S2ST models, data and application issues, and performance metrics. We critically analyze the models' performance over the benchmark datasets and provide research challenges and future directions.