mlsanityData to 7 Oct 2026

CTC Alignments Improve Autoregressive Translation

arXiv 2210.05200 ↗First posted 11 Oct 2022cs.CL cs.SD eess.AScited by 25

Not on the map. Why?

Abstract (arXiv)

Connectionist Temporal Classification (CTC) is a widely used approach for automatic speech recognition (ASR) that performs conditionally independent monotonic alignment. However for translation, CTC exhibits clear limitations due to the contextual and non-monotonic nature of the task and thus lags behind attentional decoder approaches in terms of translation quality. In this work, we argue that CTC does in fact make sense for translation if applied in a joint CTC/attention framework wherein CTC's core properties can counteract several key weaknesses of pure-attention models during training and decoding. To validate this conjecture, we modify the Hybrid CTC/Attention model originally proposed for ASR to support text-to-text translation (MT) and speech-to-text translation (ST). Our proposed joint CTC/attention models outperform pure-attention baselines across six benchmark translation tasks.

Key facts. Cited by 25 papers in the map (data through 7 Oct 2026). Builds on 48 of its 80 references that are in the map. First posted on arXiv 11 Oct 2022.

Builds on

48 of 80 references.

References · 48

  1. 2017Attention Is All You Needcited by 59,932
  2. 2014Neural Machine Translation by Jointly Learning to Align and Translatecited by 4,530
  3. 2020wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representationscited by 3,853
  4. 2021HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Unitscited by 2,301
  5. 2018SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processingcited by 1,973
  6. 2020Conformer: Convolution-augmented Transformer for Speech Recognitioncited by 1,838
  7. 2018A Call for Clarity in Reporting BLEU Scorescited by 1,790
  8. 2015Sequence Level Training with Recurrent Neural Networkscited by 568
  9. 2018ESPnet: End-to-End Speech Processing Toolkitcited by 554
  10. 2017Convolutional Sequence to Sequence Learningcited by 550
  11. 2015Listen, Attend and Spellcited by 542
  12. 2015Attention-Based Models for Speech Recognitioncited by 400
36 more
  1. 2016Joint CTC-Attention based End-to-End Speech Recognition using Multi-task Learningcited by 254
  2. 2021Composable Sparse Fine-Tuning for Cross-Lingual Transfercited by 144
  3. 2020Recent Developments on ESPnet Toolkit Boosted by Conformercited by 132
  4. 2019Understanding Knowledge Distillation in Non-autoregressive Machine Translationcited by 121
  5. 2018Correcting Length Bias in Neural Machine Translationcited by 119
  6. 2019Europarl-ST: A Multilingual Corpus For Speech Translation Of Parliamentary Debatescited by 114
  7. 2020Glancing Transformer for Non-Autoregressive Neural Machine Translationcited by 103
  8. 2021The Multilingual TEDx Corpus for Speech Recognition and Translationcited by 101
  9. 2020Fully Non-autoregressive Neural Machine Translation: Tricks of the Tradecited by 99
  10. 2020Non-Autoregressive Machine Translation with Latent Alignmentscited by 96
  11. 2020ESPnet-ST: All-in-One Speech Translation Toolkitcited by 82
  12. 2018End-to-End Non-Autoregressive Neural Machine Translation with Connectionist Temporal Classificationcited by 72
  13. 2020Imputer: Sequence Modelling via Imputation and Dynamic Programmingcited by 65
  14. 2020Aligned Cross Entropy for Non-Autoregressive Machine Translationcited by 65
  15. 2019Two-Pass End-to-End Speech Recognitioncited by 62
  16. 2020Dual-decoder Transformer for Joint Automatic Speech Recognition and Multilingual Speech Translationcited by 54
  17. 2021CTC-based Compression for Direct Speech Translationcited by 45
  18. 2019compare-mt: A Tool for Holistic Comparison of Language Generation Systemscited by 43
  19. 2020Best-First Beam Searchcited by 40
  20. 2021Non-Autoregressive Translation with Layer-Wise Prediction and Deep Supervisioncited by 40
  21. 2014First-Pass Large Vocabulary Continuous Speech Recognition using Bi-Directional Recurrent DNNscited by 27
  22. 2022Revisiting End-to-End Speech-to-Text Translation From Scratchcited by 26
  23. 2019Enhancing Machine Translation with Dependency-Aware Self-Attentioncited by 24
  24. 2021Searchable Hidden Intermediates for End-to-End Models of Decomposable Sequence Taskscited by 24
  25. 2021Investigating the Reordering Capability in CTC-based Non-Autoregressive End-to-End Speech Translationcited by 23
  26. 2018Hierarchical Multi Task Learning With CTCcited by 20
  27. 2020On Long-Tailed Phenomena in Neural Machine Translationcited by 20
  28. 2021Regularizing End-to-End Speech Translation with Triangular Decomposition Agreementcited by 16
  29. 2021Hierarchical Conditional End-to-End ASR with CTC and Multi-Granular Subword Unitscited by 13
  30. 2019Neural Machine Translation with Explicit Phrase Alignmentcited by 12
  31. 2022Blockwise Streaming Transformer for Spoken Language Understanding and Simultaneous Speech Translationcited by 11
  32. 2021Non-autoregressive End-to-end Speech Translation with Parallel Autoregressive Rescoringcited by 9
  33. 2021Fast-MD: Fast Multi-Decoder End-to-End Speech Translation with Non-Autoregressive Hidden Intermediatescited by 9
  34. 2022LegoNN: Building Modular Encoder-Decoder Modelscited by 9
  35. 2021Can Latent Alignments Improve Autoregressive Machine Translation?cited by 3
  36. 2019Source Dependency-Aware Transformer with Supervised Self-Attentioncited by 2

Built on by

Cited by year

Citing papers per year: 2022: 4; 2023: 13; 2024: 4; 2025: 4.

25 papers.

Sort:

  1. 2023Exploring Speech Recognition, Translation, and Understanding with Discrete Speech Units: A Comparative Studycited by 42
  2. 2024OWSM-CTC: An Open Encoder-Only Speech Foundation Model for Speech Recognition, Translation, and Language Identificationcited by 34
  3. 2023Improving Massively Multilingual ASR With Auxiliary CTC Objectivescited by 31
  4. 2023Hybrid Transducer and Attention based Encoder-Decoder Modeling for Speech-to-Text Taskscited by 20
  5. 2023A Comparative Study on E-Branchformer vs Conformer in Speech Recognition, Translation, and Understanding Taskscited by 13
  6. 2022Bayes risk CTC: Controllable CTC alignment in Sequence-to-Sequence taskscited by 12
  7. 2022LAMASSU: Streaming Language-Agnostic Multilingual Speech Recognition and Translation Using Neural Transducerscited by 10
  8. 2023ESPnet-ST-v2: Multipurpose Spoken Language Translation Toolkitcited by 10
  9. 2023CTC-based Non-autoregressive Speech Translationcited by 9
  10. 2023Incremental Blockwise Beam Search for Simultaneous Speech Translation with Controllable Quality-Latency Tradeoffcited by 8
  11. 20224D ASR: Joint modeling of CTC, Attention, Transducer, and Mask-Predict decoderscited by 7
  12. 2023Long-Form End-to-End Speech Translation via Latent Alignment Segmentationcited by 6
13 more
  1. 2023BA-SOT: Boundary-Aware Serialized Output Training for Multi-Talker ASRcited by 5
  2. 2022Align, Write, Re-order: Explainable End-to-End Speech Translation via Operation Sequence Generationcited by 4
  3. 2024Joint Beam Search Integrating CTC, Attention, and Transducer Decoderscited by 4
  4. 2024Longer is (Not Necessarily) Stronger: Punctuated Long-Sequence Training for Enhanced Speech Recognition and Translationcited by 4
  5. 2023Lego-Features: Exporting modular encoder features for streaming and deliberation ASRcited by 3
  6. 2023Strategies for improving low resource speech to text translation relying on pre-trained ASR modelscited by 3
  7. 2023Enhancing End-to-End Conversational Speech Translation Through Target Language Context Utilizationcited by 3
  8. 2023Cross-Modal Multi-Tasking for Speech-to-Text Translation via Hard Parameter Sharingcited by 3
  9. 2025FAMA: The First Large-Scale Open-Science Speech Foundation Model for English and Italiancited by 2
  10. 2025Enhanced Hybrid Transducer and Attention Encoder Decoder with Text Datacited by 1
  11. 2025Cross-Attention is Half Explanation in Speech-to-Text Modelscited by 1
  12. 2024ESPnet-EZ: Python-only ESPnet for Easy Fine-tuning and Integrationcited by 0
  13. 2025A Differentiable Alignment Framework for Sequence-to-Sequence Modeling via Optimal Transportcited by 0