mlsanityData to 7 Oct 2026

CTC-based Non-autoregressive Speech Translation

arXiv 2305.17358 ↗First posted 27 May 2023cs.CLcited by 9

Not on the map. Why?

Abstract (arXiv)

Combining end-to-end speech translation (ST) and non-autoregressive (NAR) generation is promising in language and speech processing for their advantages of less error propagation and low latency. In this paper, we investigate the potential of connectionist temporal classification (CTC) for non-autoregressive speech translation (NAST). In particular, we develop a model consisting of two encoders that are guided by CTC to predict the source and target texts, respectively. Introducing CTC into NAST on both language sides has obvious challenges: 1) the conditional independent generation somewhat breaks the interdependency among tokens, and 2) the monotonic alignment assumption in standard CTC does not hold in translation tasks. In response, we develop a prediction-aware encoding approach and a cross-layer attention approach to address these issues. We also use curriculum learning to improve convergence of training. Experiments on the MuST-C ST benchmarks show that our NAST model achieves an average BLEU score of 29.5 with a speed-up of 5.67×, which is comparable to the autoregressive counterpart and even outperforms the previous best result of 0.9 BLEU points.

Key facts. Cited by 9 papers in the map (data through 7 Oct 2026). Builds on 46 of its 66 references that are in the map. First posted on arXiv 27 May 2023.

Builds on

46 of 66 references.

References · 46

  1. 2018A Call for Clarity in Reporting BLEU Scorescited by 1,790
  2. 2019fairseq: A Fast, Extensible Toolkit for Sequence Modelingcited by 1,486
  3. 2016Sequence-Level Knowledge Distillationcited by 761
  4. 2017Non-Autoregressive Neural Machine Translationcited by 457
  5. 2018Deterministic Non-Autoregressive Neural Sequence Modeling by Iterative Refinementcited by 194
  6. 2020fairseq S2T: Fast Speech-to-Text Modeling with fairseqcited by 174
  7. 2019Insertion Transformer: Flexible Sequence Generation via Insertion Operationscited by 123
  8. 2017Sequence-to-Sequence Models Can Directly Translate Foreign Speechcited by 121
  9. 2016Listen and Translate: A Proof of Concept for End-to-End Speech-to-Text Translationcited by 113
  10. 2020Glancing Transformer for Non-Autoregressive Neural Machine Translationcited by 103
  11. 2020Fully Non-autoregressive Neural Machine Translation: Tricks of the Tradecited by 99
  12. 2020Non-Autoregressive Machine Translation with Latent Alignmentscited by 96
34 more
  1. 2020ESPnet-ST: All-in-One Speech Translation Toolkitcited by 82
  2. 2020Mask CTC: Non-Autoregressive End-to-End ASR with CTC and Mask Predictcited by 75
  3. 2018End-to-End Non-Autoregressive Neural Machine Translation with Connectionist Temporal Classificationcited by 72
  4. 2020Imputer: Sequence Modelling via Imputation and Dynamic Programmingcited by 65
  5. 2021Relaxing the Conditional Independence Assumption of CTC-based ASR by Conditioning on Intermediate Predictionscited by 61
  6. 2022Cross-modal Contrastive Learning for Speech Translationcited by 61
  7. 2019Latent-Variable Non-Autoregressive Neural Machine Translation with Deterministic Inference Using a Delta Posteriorcited by 58
  8. 2020Curriculum Pre-training for End-to-End Speech Translationcited by 57
  9. 2022STEMM: Self-learning with Speech-text Manifold Mixup for Speech Translationcited by 57
  10. 2020Bridging the Modality Gap for Speech-to-Text Translationcited by 55
  11. 2018Tied Multitask Learning for Neural Speech Translationcited by 54
  12. 2019Non-Autoregressive Machine Translation with Auxiliary Regularizationcited by 52
  13. 2022Squeezeformer: An Efficient Transformer for Automatic Speech Recognitioncited by 50
  14. 2021Stacked Acoustic-and-Textual Encoding: Integrating the Pre-trained Models into Speech Translation Encoderscited by 48
  15. 2021End-to-end Speech Translation via Cross-modal Progressive Trainingcited by 47
  16. 2018Modeling Localness for Self-Attention Networkscited by 43
  17. 2021Non-Autoregressive Translation with Layer-Wise Prediction and Deep Supervisioncited by 40
  18. 2018Non-Autoregressive Neural Machine Translation with Enhanced Decoder Inputcited by 39
  19. 2019Bridging the Gap between Pre-Training and Fine-Tuning for End-to-End Speech Translationcited by 39
  20. 2019Guiding Non-Autoregressive Neural Machine Translation Decoding with Reordering Informationcited by 37
  21. 2021Rejuvenating Low-Frequency Words: Making the Most of Parallel Data in Non-Autoregressive Translationcited by 34
  22. 2020Improved Mask-CTC for Non-Autoregressive End-to-End ASRcited by 32
  23. 2021A Comparative Study on Non-Autoregressive Modelings for Speech-to-Text Generationcited by 27
  24. 2022Revisiting End-to-End Speech-to-Text Translation From Scratchcited by 26
  25. 2022CTC Alignments Improve Autoregressive Translationcited by 25
  26. 2021AligNART: Non-autoregressive Neural Machine Translation by Jointly Learning to Estimate Alignment and Translatecited by 24
  27. 2020NeurST: Neural Speech Translation Toolkitcited by 23
  28. 2021Investigating the Reordering Capability in CTC-based Non-Autoregressive End-to-End Speech Translationcited by 23
  29. 2022M3ST: Mix at Three Levels for Speech Translationcited by 21
  30. 2022textitlatent-GLAT: Glancing at Latent Variables for Parallel Text Generationcited by 20
  31. 2020Learning to Recover from Multi-Modality Errors for Non-Autoregressive Neural Machine Translationcited by 19
  32. 2020Orthros: Non-autoregressive End-to-end Speech Translation with Dual-decodercited by 17
  33. 2021Non-autoregressive End-to-end Speech Translation with Parallel Autoregressive Rescoringcited by 9
  34. 2022Improving End-to-end Speech Translation by Leveraging Auxiliary Speech and Text Datacited by 9

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 2; 2024: 5; 2025: 0; 2026: 2.

9 papers.

  1. 2024OWSM-CTC: An Open Encoder-Only Speech Foundation Model for Speech Recognition, Translation, and Language Identificationcited by 34
  2. 2023Non-autoregressive Streaming Transformer for Simultaneous Translationcited by 12
  3. 2024HAINAN: Fast and Accurate Transducer for Hybrid-Autoregressive ASRcited by 2
  4. 2024CTC-based Non-autoregressive Textless Speech-to-Speech Translationcited by 1
  5. 2024A Modular-based Strategy for Mitigating Gradient Conflicts in Simultaneous Speech Translationcited by 1
  6. 2023Bridging the Gaps of Both Modality and Language: Synchronous Bilingual CTC for Speech Translation and Speech Recognitioncited by 0
  7. 2024Revisiting Interpolation Augmentation for Speech-to-Text Generationcited by 0
  8. 2026Reconsidering Positional Supervision in Masked Diffusion Language Model Trainingcited by 0
  9. 2026A Generalized Formalism of Auto-Regressive Decoding for Speech Processingcited by 0