mlsanityData to 14 Oct 2026

Direct Simultaneous Speech-to-Speech Translation with Variational Monotonic Multihead Attention

arXiv 2110.08250 ↗First posted 15 Oct 2021cs.CL cs.SD eess.AScited by 9

Not on the map. Why?

Abstract (arXiv)

We present a direct simultaneous speech-to-speech translation (Simul-S2ST) model, Furthermore, the generation of translation is independent from intermediate text representations. Our approach leverages recent progress on direct speech-to-speech translation with discrete units, in which a sequence of discrete representations, instead of continuous spectrogram features, learned in an unsupervised manner, are predicted from the model and passed directly to a vocoder for speech synthesis on-the-fly. We also introduce the variational monotonic multihead attention (V-MMA), to handle the challenge of inefficient policy learning in speech simultaneous translation. The simultaneous policy then operates on source speech features and target discrete units. We carry out empirical studies to compare cascaded and direct approach on the Fisher Spanish-English and MuST-C English-Spanish datasets. Direct simultaneous model is shown to outperform the cascaded model by achieving a better tradeoff between translation quality and latency.

Key facts. Cited by 9 papers in the map (data through 14 Oct 2026). Builds on 24 of its 30 references that are in the map. First posted on arXiv 15 Oct 2021.

Builds on

24 of 30 references.

References · 24

  1. 2017Attention Is All You Needcited by 60,361
  2. 2021HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Unitscited by 2,342
  3. 2020HiFi-GAN: Generative Adversarial Networks for Efficient and High Fidelity Speech Synthesiscited by 1,154
  4. 2020FastSpeech 2: Fast and High-Quality End-to-End Text to Speechcited by 717
  5. 2020fairseq S2T: Fast Speech-to-Text Modeling with fairseqcited by 176
  6. 2021Direct speech-to-speech translation with discrete unitscited by 149
  7. 2019Direct speech-to-speech translation with a sequence-to-sequence modelcited by 118
  8. 2020Emformer: Efficient Memory Transformer Based Acoustic Model For Low Latency Streaming Speech Recognitioncited by 113
  9. 2019Monotonic Infinite Lookback Attention for Simultaneous Machine Translationcited by 104
  10. 2016Learning to Translate in Real-time with Neural Machine Translationcited by 97
  11. 2020SimulEval: An Evaluation Toolkit for Simultaneous Translationcited by 94
  12. 2019Monotonic Multihead Attentioncited by 93
12 more
  1. 2016Can neural machine translation do simultaneous translation?cited by 86
  2. 2017Online and Linear-Time Attention by Enforcing Monotonic Alignmentscited by 85
  3. 2017Monotonic Chunkwise Attentioncited by 83
  4. 2020SimulMT to SimulST: Adapting Simultaneous Text Translation to End-to-End Simultaneous Speech Translationcited by 81
  5. 2018Incremental Decoding and Training Methods for Simultaneous Translation in Neural Machine Translationcited by 52
  6. 2020Simultaneous Translation Policies: From Fixed to Adaptivecited by 43
  7. 2020Streaming Simultaneous Speech Translation with Augmented Memory Transformercited by 30
  8. 2019Incremental Text-to-Speech Synthesis with Prefix-to-Prefix Frameworkcited by 15
  9. 2020Simultaneous Speech-to-Speech Translation System with Neural Incremental ASR, MT, and TTScited by 12
  10. 2020Fluent and Low-latency Simultaneous Speech-to-Speech Translation with Self-adaptive Trainingcited by 10
  11. 2021High Fidelity Speech Regeneration with Application to Speech Enhancementcited by 10
  12. 2021Alternate Endings: Improving Prosody for Incremental Neural TTS with Predicted Future Text Inputcited by 7

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2022: 3; 2023: 2; 2024: 3; 2025: 0; 2026: 1.

9 papers.

  1. 2023AudioPaLM: A Large Language Model That Can Speak and Listencited by 343
  2. 2022CVSS Corpus and Massively Multilingual Speech-to-Speech Translationcited by 71
  3. 2022Leveraging unsupervised and weakly-supervised data to improve direct speech-to-speech translationcited by 20
  4. 2022Textless Direct Speech-to-Speech Translation with Discrete Speech Representationcited by 19
  5. 2024Direct Speech-to-Speech Neural Machine Translation: A Surveycited by 6
  6. 2024Overcoming Non-monotonicity in Transducer-based Streaming Generationcited by 4
  7. 2024SimulTron: On-Device Simultaneous Speech to Speech Translationcited by 1
  8. 2023Enhancing Speech-to-Speech Translation with Multiple TTS Targetscited by 0
  9. 2026X-Translator: A Real-Time Multilingual Speaker-Aware Speech-to-Speech Translation Systemcited by 0