Data to 7 Oct 2026

Direct speech-to-speech translation with a sequence-to-sequence model

arXiv 1904.06037 ↗First posted 12 Apr 2019cs.CL cs.LG cs.SDcited by 118

Not on the map. Why?

Abstract (arXiv)

We present an attention-based sequence-to-sequence neural network which can directly translate speech from one language into speech in another language, without relying on an intermediate text representation. The network is trained end-to-end, learning to map speech spectrograms into target spectrograms in another language, corresponding to the translated content (in a different canonical voice). We further demonstrate the ability to synthesize translated speech using the voice of the source speaker. We conduct experiments on two Spanish-to-English speech translation datasets, and find that the proposed model slightly underperforms a baseline cascade of a direct speech-to-text translation model and a text-to-speech synthesis model, demonstrating the feasibility of the approach on this very challenging task.

Key facts. Cited by 118 papers in the map (data through 7 Oct 2026). First posted on arXiv 12 Apr 2019.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2020: 1; 2021: 14; 2022: 23; 2023: 26; 2024: 20; 2025: 19; 2026: 15.

118 papers.

Sort:

By area:

2021VoxPopuli: A Large-Scale Multilingual Speech Corpus for Representation Learning, Semi-Supervised Learning and Interpretationcited by 455
2023AudioPaLM: A Large Language Model That Can Speak and Listencited by 333
2022BigVGAN: A Universal Neural Vocoder with Large-Scale Trainingcited by 245
2023Speak Foreign Languages with Your Own Voice: Cross-Lingual Neural Codec Language Modelingcited by 166
2021Direct speech-to-speech translation with discrete unitscited by 148
2021Textless Speech-to-Speech Translation on Real Datacited by 121
2025Step-Audio 2 Technical Reportcited by 101
2022CVSS Corpus and Massively Multilingual Speech-to-Speech Translationcited by 71
2021Translatotron 2: High-quality direct speech-to-speech translation with voice preservationcited by 69
2022UnitY: Two-pass Direct Speech-to-speech Translation with Discrete Unitscited by 60
2021Adversarial Text-to-Image Synthesis: A Reviewcited by 52
2021CTC-based Compression for Direct Speech Translationcited by 45

38 more

2022Enhanced Direct Speech-to-Speech Translation Using Self-supervised Pre-training and Data Augmentationcited by 43
2022SpeechMatrix: A Large-Scale Mined Corpus of Multilingual Speech-to-Speech Translationscited by 35
2023SLTUNET: A Simple Unified Model for Sign Language Translationcited by 35
2022TranSpeech: Speech-to-Speech Translation With Bilateral Perturbationcited by 34
2022Self-supervised language learning from raw audio: Lessons from the Zero Resource Speech Challengecited by 34
2022Speech-to-Speech Translation For A Real-world Unwritten Languagecited by 25
2021fairseq S^2: A Scalable and Integrable Speech Synthesis Toolkitcited by 24
2023EmphAssess : a Prosodic Benchmark on Assessing Emphasis Transfer in Speech-to-Speech Modelscited by 21
2022Leveraging unsupervised and weakly-supervised data to improve direct speech-to-speech translationcited by 20
2022WaveFit: An Iterative and Non-autoregressive Neural Vocoder based on Fixed-Point Iterationcited by 20
2024StreamSpeech: Simultaneous Speech-to-Speech Translation with Multi-task Learningcited by 20
2022Textless Direct Speech-to-Speech Translation with Discrete Speech Representationcited by 19
2023PolyVoice: Language Models for Speech to Speech Translationcited by 19
2024MeLFusion: Synthesizing Music from Image and Language Cues using Diffusion Modelscited by 19
2022T-Modules: Translation Modules for Zero-Shot Cross-Modal Machine Translationcited by 16
2022BLASER: A Text-Free Speech-to-Speech Translation Evaluation Metriccited by 16
2025High-Fidelity Simultaneous Speech-To-Speech Translationcited by 15
2023AV2AV: Direct Audio-Visual Speech to Audio-Visual Speech Translation with Unified Audio-Visual Speech Representationcited by 14
2024Translatotron-V(ison): An End-to-End Model for In-Image Machine Translationcited by 14
2025MTalk-Bench: Evaluating Speech-to-Speech Models in Multi-Turn Dialogues via Arena-style and Rubrics Protocolscited by 14
2022Leveraging Pseudo-labeled Data to Improve Direct Speech-to-Speech Translationcited by 12
2022Simple and Effective Unsupervised Speech Translationcited by 12
2023Translatotron 3: Speech to Speech Translation with Monolingual Datacited by 12
2024SpeechPrompt: Prompting Speech Language Models for Speech Processing Taskscited by 12
2021Rethinking End-to-End Evaluation of Decomposable Tasks: A Case Study on Spoken Language Understandingcited by 11
2023Textless Unit-to-Unit training for Many-to-Many Multilingual Speech-to-Speech Translationcited by 11
2020Learning Coupled Policies for Simultaneous Machine Translation using Imitation Learningcited by 10
2021Assessing Evaluation Metrics for Speech-to-Speech Translationcited by 10
2023A Holistic Cascade System, benchmark, and Human Evaluation Protocol for Expressive Speech-to-Speech Translationcited by 10
2023ESPnet-ST-v2: Multipurpose Spoken Language Translation Toolkitcited by 10
2024A Non-autoregressive Generation Framework for End-to-End Simultaneous Speech-to-Speech Translationcited by 10
2021Direct Simultaneous Speech-to-Speech Translation with Variational Monotonic Multihead Attentioncited by 9
2022Joint Pre-Training with Speech and Bilingual Text for Direct Speech to Speech Translationcited by 9
2023DASpeech: Directed Acyclic Transformer for Fast and High-quality Speech-to-Speech Translationcited by 9
2024SeamlessExpressiveLM: Speech Language Model for Expressive Speech-to-Speech Translation with Chain-of-Thoughtcited by 9
2025Seed LiveInterpret 2.0: End-to-end Simultaneous Speech-to-speech Translation with Your Voicecited by 9
2021Towards the evaluation of automatic simultaneous speech translation from a communicative perspectivecited by 8
2022Improving Speech-to-Speech Translation Through Unlabeled Textcited by 8

50 most cited of 118 All 118 (JSON)