mlsanityData to 7 Oct 2026

CTC-based Non-autoregressive Textless Speech-to-Speech Translation

arXiv 2406.07330 ↗First posted 11 Jun 2024cs.CL cs.AI cs.SDcited by 1

Not on the map. Why?

Abstract (arXiv)

Direct speech-to-speech translation (S2ST) has achieved impressive translation quality, but it often faces the challenge of slow decoding due to the considerable length of speech sequences. Recently, some research has turned to non-autoregressive (NAR) models to expedite decoding, yet the translation quality typically lags behind autoregressive (AR) models significantly. In this paper, we investigate the performance of CTC-based NAR models in S2ST, as these models have shown impressive results in machine translation. Experimental results demonstrate that by combining pretraining, knowledge distillation, and advanced NAR training techniques such as glancing training and non-monotonic latent alignments, CTC-based NAR models achieve translation quality comparable to the AR model, while preserving up to 26.81× decoding speedup.

Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Builds on 25 of its 35 references that are in the map. First posted on arXiv 11 Jun 2024.

Builds on

25 of 35 references.

References · 25

  1. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  2. 2021HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Unitscited by 2,301
  3. 2020Conformer: Convolution-augmented Transformer for Speech Recognitioncited by 1,838
  4. 2019Transformer-XL: Attentive Language Models Beyond a Fixed-Length Contextcited by 1,830
  5. 2020HiFi-GAN: Generative Adversarial Networks for Efficient and High Fidelity Speech Synthesiscited by 1,139
  6. 2016Sequence-Level Knowledge Distillationcited by 761
  7. 2017Non-Autoregressive Neural Machine Translationcited by 457
  8. 2021Direct speech-to-speech translation with discrete unitscited by 148
  9. 2019Understanding Knowledge Distillation in Non-autoregressive Machine Translationcited by 121
  10. 2021Textless Speech-to-Speech Translation on Real Datacited by 121
  11. 2019Direct speech-to-speech translation with a sequence-to-sequence modelcited by 118
  12. 2020Glancing Transformer for Non-Autoregressive Neural Machine Translationcited by 103
13 more
  1. 2018End-to-End Non-Autoregressive Neural Machine Translation with Connectionist Temporal Classificationcited by 72
  2. 2021Translatotron 2: High-quality direct speech-to-speech translation with voice preservationcited by 69
  3. 2022UnitY: Two-pass Direct Speech-to-speech Translation with Discrete Unitscited by 60
  4. 2022STEMM: Self-learning with Speech-text Manifold Mixup for Speech Translationcited by 57
  5. 2022Speech-to-Speech Translation For A Real-world Unwritten Languagecited by 25
  6. 2023CMOT: Cross-modal Mixup via Optimal Transport for Speech Translationcited by 22
  7. 2022One Reference Is Not Enough: Diverse Distillation with Reference Selection for Non-Autoregressive Translationcited by 20
  8. 2022Non-Monotonic Latent Alignments for CTC-Based Non-Autoregressive Machine Translationcited by 16
  9. 2023Understanding and Bridging the Modality Gap for Speech Translationcited by 15
  10. 2021Non-autoregressive End-to-end Speech Translation with Parallel Autoregressive Rescoringcited by 9
  11. 2023Back Translation for Speech-to-text Translation Without Transcriptscited by 9
  12. 2023CTC-based Non-autoregressive Speech Translationcited by 9
  13. 2023Duplex Diffusion Models Improve Speech-to-Speech Translationcited by 5

Built on by

Cited by year

Citing papers per year: 2025: 1.

1 paper.

  1. 2025Transformer-Encoder Trees for Efficient Multilingual Machine Translation and Speech Translationcited by 0