mlsanityData to 7 Oct 2026

Fast-MD: Fast Multi-Decoder End-to-End Speech Translation with Non-Autoregressive Hidden Intermediates

arXiv 2109.12804 ↗First posted 27 Sep 2021eess.AS cs.CL cs.SDcited by 9

Not on the map. Why?

Abstract (arXiv)

The multi-decoder (MD) end-to-end speech translation model has demonstrated high translation quality by searching for better intermediate automatic speech recognition (ASR) decoder states as hidden intermediates (HI). It is a two-pass decoding model decomposing the overall task into ASR and machine translation sub-tasks. However, the decoding speed is not fast enough for real-world applications because it conducts beam search for both sub-tasks during inference. We propose Fast-MD, a fast MD model that generates HI by non-autoregressive (NAR) decoding based on connectionist temporal classification (CTC) outputs followed by an ASR decoder. We investigated two types of NAR HI: (1) parallel HI by using an autoregressive Transformer ASR decoder and (2) masked HI by using Mask-CTC, which combines CTC and the conditional masked language model. To reduce a mismatch in the ASR decoder between teacher-forcing during training and conditioning on CTC outputs during testing, we also propose sampling CTC outputs during training. Experimental evaluations on three corpora show that Fast-MD achieved about 2x and 4x faster decoding speed than that of the na"ive MD model on GPU and CPU with comparable translation quality. Adopting the Conformer encoder and intermediate CTC loss further boosts its quality without sacrificing decoding speed.

Key facts. Cited by 9 papers in the map (data through 7 Oct 2026). Builds on 50 of its 72 references that are in the map. First posted on arXiv 27 Sep 2021.

Builds on

50 of 72 references.

References · 50

  1. 2017Attention Is All You Needcited by 59,932
  2. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  3. 2015Neural Machine Translation of Rare Words with Subword Unitscited by 2,898
  4. 2018SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processingcited by 1,973
  5. 2020Conformer: Convolution-augmented Transformer for Speech Recognitioncited by 1,838
  6. 2018A Call for Clarity in Reporting BLEU Scorescited by 1,790
  7. 2019SpecAugment: A Simple Data Augmentation Method for Automatic Speech Recognitioncited by 1,355
  8. 2019A Comparative Study on Transformer vs RNN in Speech Applicationscited by 210
  9. 2019pyannote.audio: neural building blocks for speaker diarizationcited by 175
  10. 2020fairseq S2T: Fast Speech-to-Text Modeling with fairseqcited by 174
  11. 2020Recent Developments on ESPnet Toolkit Boosted by Conformercited by 132
  12. 2017Sequence-to-Sequence Models Can Directly Translate Foreign Speechcited by 121
38 more
  1. 2021Intermediate Loss Regularization for CTC-based Speech Recognitioncited by 95
  2. 2020ESPnet-ST: All-in-One Speech Translation Toolkitcited by 82
  3. 2020Mask CTC: Non-Autoregressive End-to-End ASR with CTC and Mask Predictcited by 75
  4. 2018Pre-training on high-resource speech recognition improves low-resource speech-to-text translationcited by 70
  5. 2018End-to-End Automatic Speech Translation of Audiobookscited by 65
  6. 2021Relaxing the Conditional Independence Assumption of CTC-based ASR by Conditioning on Intermediate Predictionscited by 61
  7. 2019End-to-End Speech Translation with Knowledge Distillationcited by 58
  8. 2020Curriculum Pre-training for End-to-End Speech Translationcited by 57
  9. 2020Bridging the Modality Gap for Speech-to-Text Translationcited by 55
  10. 2018Tied Multitask Learning for Neural Speech Translationcited by 54
  11. 2020Dual-decoder Transformer for Joint Automatic Speech Recognition and Multilingual Speech Translationcited by 54
  12. 2021Stacked Acoustic-and-Textual Encoding: Integrating the Pre-trained Models into Speech Translation Encoderscited by 48
  13. 2021CTC-based Compression for Direct Speech Translationcited by 45
  14. 2019A Comparative Study on End-to-end Speech to Text Translationcited by 40
  15. 2019Bridging the Gap between Pre-Training and Fine-Tuning for End-to-End Speech Translationcited by 39
  16. 2021RealTranS: End-to-End Simultaneous Speech Translation with Convolutional Weighted-Shrinking Transformercited by 39
  17. 2020Unified Streaming and Non-streaming Two-pass End-to-end Model for Speech Recognitioncited by 37
  18. 2018Augmenting Librispeech with French Translations: A Multimodal Corpus for Direct Speech Translation Evaluationcited by 34
  19. 2020Improved Mask-CTC for Non-Autoregressive End-to-End ASRcited by 32
  20. 2019Attention-Passing Models for Robust and Data-Efficient End-to-End Speech Translationcited by 30
  21. 2020Consecutive Decoding for Speech-to-text Translationcited by 28
  22. 2021Source and Target Bidirectional Knowledge Distillation for End-to-end Speech Translationcited by 27
  23. 2020Adaptive Feature Selection for End-to-End Speech Translationcited by 25
  24. 2019Deja-vu: Double Feature Presentation and Iterated Loss in Deep Transformer Networkscited by 24
  25. 2020CASS-NAT: CTC Alignment-based Single Step Non-autoregressive Transformer for Speech Recognitioncited by 24
  26. 2020Non-Autoregressive Transformer ASR with CTC-Enhanced Decoder Inputcited by 24
  27. 2021Searchable Hidden Intermediates for End-to-End Models of Decomposable Sequence Taskscited by 24
  28. 2020Relative Positional Encoding for Speech Recognition and Direct Translationcited by 23
  29. 2020NeurST: Neural Speech Translation Toolkitcited by 23
  30. 2021Investigating the Reordering Capability in CTC-based Non-Autoregressive End-to-End Speech Translationcited by 23
  31. 2019Synchronous Speech Recognition and Speech-to-Text Translation with Interactive Decodingcited by 21
  32. 2021Pushing the Limits of Non-Autoregressive Speech Recognitioncited by 18
  33. 2020Orthros: Non-autoregressive End-to-end Speech Translation with Dual-decodercited by 17
  34. 2021Beyond Voice Activity Detection: Hybrid Audio Segmentation for Direct Speech Translationcited by 17
  35. 2020Contextualized Translation of Automatically Segmented Speechcited by 12
  36. 2021WNARS: WFST based Non-autoregressive Streaming End-to-End Speech Recognitioncited by 10
  37. 2019Enforcing Encoder-Decoder Modularity in Sequence-to-Sequence Modelscited by 7
  38. 2020One In A Hundred: Select The Best Predicted Sequence from Numerous Candidates for Streaming Speech Recognitioncited by 4

Built on by

Cited by year

Citing papers per year: 2022: 4; 2023: 2; 2024: 2; 2025: 1.

9 papers.

  1. 2022A Survey on Non-Autoregressive Generation for Neural Machine Translation and Beyondcited by 64
  2. 2022UnitY: Two-pass Direct Speech-to-speech Translation with Discrete Unitscited by 60
  3. 2022CTC Alignments Improve Autoregressive Translationcited by 25
  4. 2023Recent Advances in Direct Speech-to-text Translationcited by 15
  5. 2022Blockwise Streaming Transformer for Spoken Language Understanding and Simultaneous Speech Translationcited by 11
  6. 2024Automatic Speech Recognition with BERT and CTC Transformers: A Reviewcited by 8
  7. 2024Label-Synchronous Neural Transducer for E2E Simultaneous Speech Translationcited by 5
  8. 2025When End-to-End is Overkill: Rethinking Cascaded Speech-to-Text Translationcited by 4
  9. 2023Decoupled Structure for Improved Adaptability of End-to-End Modelscited by 3