mlsanityData to 7 Oct 2026

Intermediate Loss Regularization for CTC-based Speech Recognition

arXiv 2102.03216 ↗First posted 5 Feb 2021eess.AS cs.CL cs.SDcited by 95

Not on the map. Why?

Abstract (arXiv)

We present a simple and efficient auxiliary loss function for automatic speech recognition (ASR) based on the connectionist temporal classification (CTC) objective. The proposed objective, an intermediate CTC loss, is attached to an intermediate layer in the CTC encoder network. This intermediate CTC loss well regularizes CTC training and improves the performance requiring only small modification of the code and small and no overhead during training and inference, respectively. In addition, we propose to combine this intermediate CTC loss with stochastic depth training, and apply this combination to a recently proposed Conformer network. We evaluate the proposed method on various corpora, reaching word error rate (WER) 9.9% on the WSJ corpus and character error rate (CER) 5.2% on the AISHELL-1 corpus respectively, based on CTC greedy search without a language model. Especially, the AISHELL-1 task is comparable to other state-of-the-art ASR systems based on auto-regressive decoder with beam search.

Key facts. Cited by 95 papers in the map (data through 7 Oct 2026). Builds on 32 of its 42 references that are in the map. First posted on arXiv 5 Feb 2021.

Builds on

32 of 42 references.

References · 32

  1. 2017Attention Is All You Needcited by 59,932
  2. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  3. 2016Layer Normalizationcited by 5,579
  4. 2020wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representationscited by 3,853
  5. 2018SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processingcited by 1,973
  6. 2020Conformer: Convolution-augmented Transformer for Speech Recognitioncited by 1,838
  7. 2019SpecAugment: A Simple Data Augmentation Method for Automatic Speech Recognitioncited by 1,355
  8. 2016Deep Networks with Stochastic Depthcited by 973
  9. 2018ESPnet: End-to-End Speech Processing Toolkitcited by 554
  10. 2015Listen, Attend and Spellcited by 542
  11. 2017Non-Autoregressive Neural Machine Translationcited by 457
  12. 2015Attention-Based Models for Speech Recognitioncited by 400
20 more
  1. 2017AISHELL-1: An Open-Source Mandarin Speech Corpus and A Speech Recognition Baselinecited by 364
  2. 2019A Comparative Study on Transformer vs RNN in Speech Applicationscited by 210
  3. 2019QuartzNet: Deep Automatic Speech Recognition with 1D Time-Channel Separable Convolutionscited by 93
  4. 2019FlowSeq: Non-Autoregressive Conditional Sequence Generation with Generative Flowcited by 89
  5. 2020Mask CTC: Non-Autoregressive End-to-End ASR with CTC and Mask Predictcited by 75
  6. 2015EESEN: End-to-End Speech Recognition using Deep RNN Models and WFST-based Decodingcited by 73
  7. 2020Imputer: Sequence Modelling via Imputation and Dynamic Programmingcited by 65
  8. 2019Latent-Variable Non-Autoregressive Neural Machine Translation with Deterministic Inference Using a Delta Posteriorcited by 58
  9. 2019Very Deep Self-Attention Networks for End-to-End Speech Recognitioncited by 46
  10. 2020Align-Refine: Non-Autoregressive Speech Recognition via Iterative Realignmentcited by 37
  11. 2017Multitask Learning with Low-Level Auxiliary Tasks for Encoder-Decoder Based Speech Recognitioncited by 29
  12. 2020Improving RNN Transducer Based ASR with Auxiliary Taskscited by 27
  13. 2019Deja-vu: Double Feature Presentation and Iterated Loss in Deep Transformer Networkscited by 24
  14. 2020Spike-Triggered Non-Autoregressive Transformer for End-to-End Speech Recognitioncited by 24
  15. 2020Scaling Up Online Speech Recognition Using ConvNetscited by 16
  16. 2020SAN-M: Memory Equipped Self-Attention for End-to-End Speech Recognitioncited by 16
  17. 2018Hierarchical Multitask Learning for CTC-based Speech Recognitioncited by 14
  18. 2018Optimal Completion Distillation for Sequence Learningcited by 14
  19. 2020Insertion-Based Modeling for End-to-End Automatic Speech Recognitioncited by 13
  20. 2020Self-and-Mixed Attention Decoder with Deep Acoustic Structure for Transformer-based LVCSRcited by 3

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 10; 2022: 22; 2023: 24; 2024: 21; 2025: 14; 2026: 4.

95 papers.

Sort:

By area:

  1. 2023End-to-End Speech Recognition: A Surveycited by 127
  2. 2022Visual Speech Recognition for Multiple Languages in the Wildcited by 94
  3. 2022A Survey on Non-Autoregressive Generation for Neural Machine Translation and Beyondcited by 64
  4. 2021Relaxing the Conditional Independence Assumption of CTC-based ASR by Conditioning on Intermediate Predictionscited by 61
  5. 2024OWSM-CTC: An Open Encoder-Only Speech Foundation Model for Speech Recognition, Translation, and Language Identificationcited by 34
  6. 2023Improving Massively Multilingual ASR With Auxiliary CTC Objectivescited by 31
  7. 2021A Comparative Study on Non-Autoregressive Modelings for Speech-to-Text Generationcited by 27
  8. 2023Audio-Visual Efficient Conformer for Robust Speech Recognitioncited by 22
  9. 2022Improving CTC-based speech recognition via knowledge transferring from pre-trained language modelscited by 14
  10. 2021Layer Pruning on Demand with Intermediate CTCcited by 13
  11. 2021Hierarchical Conditional End-to-End ASR with CTC and Multi-Granular Subword Unitscited by 13
  12. 2022HuBERT-EE: Early Exiting HuBERT for Efficient Speech Recognitioncited by 13
38 more
  1. 2023A Comparative Study on E-Branchformer vs Conformer in Speech Recognition, Translation, and Understanding Taskscited by 13
  2. 2022A context-aware knowledge transferring strategy for CTC-based ASRcited by 12
  3. 2022Bayes risk CTC: Controllable CTC alignment in Sequence-to-Sequence taskscited by 12
  4. 2024A Study of Dropout-Induced Modality Bias on Robustness to Missing Video Frames for Audio-Visual Speech Recognitioncited by 12
  5. 2022Blockwise Streaming Transformer for Spoken Language Understanding and Simultaneous Speech Translationcited by 11
  6. 2022BERT Meets CTC: New Formulation of End-to-End Speech Recognition with Pre-trained Masked Language Modelcited by 11
  7. 2023I3D: Transformer architectures with input-dependent dynamic depth for speech recognitioncited by 10
  8. 2023ESPnet-ST-v2: Multipurpose Spoken Language Translation Toolkitcited by 10
  9. 2021Fast-MD: Fast Multi-Decoder End-to-End Speech Translation with Non-Autoregressive Hidden Intermediatescited by 9
  10. 2024Contextualized End-to-end Automatic Speech Recognition with Intermediate Biasing Losscited by 9
  11. 2021Bangla Natural Language Processing: A Comprehensive Analysis of Classical, Machine Learning, and Deep Learning Based Methodscited by 8
  12. 2021An Improved Single Step Non-autoregressive Transformer for Automatic Speech Recognitioncited by 8
  13. 2022BECTRA: Transducer-based End-to-End ASR with BERT-Enhanced Encodercited by 8
  14. 2024MLCA-AVSR: Multi-Layer Cross Attention Fusion based Audio-Visual Speech Recognitioncited by 8
  15. 2022End-to-end Speech-to-Punctuated-Text Recognitioncited by 7
  16. 20224D ASR: Joint modeling of CTC, Attention, Transducer, and Mask-Predict decoderscited by 7
  17. 2023Retrieve and Copy: Scaling ASR Personalization to Large Catalogscited by 7
  18. 2024Multilingual Audio-Visual Speech Recognition with Hybrid CTC/RNN-T Fast Conformercited by 7
  19. 2024Dynamic Encoder Size Based on Data-Driven Layer-wise Pruning for Speech Recognitioncited by 7
  20. 2021Decoupling recognition and transcription in Mandarin ASRcited by 6
  21. 2022Distilling the Knowledge of BERT for CTC-based ASRcited by 6
  22. 2022InterAug: Augmenting Noisy Intermediate Predictions for CTC-based ASRcited by 5
  23. 2022Better Intermediates Improve CTC Inferencecited by 5
  24. 2022Uconv-Conformer: High Reduction of Input Sequence Length for End-to-End Speech Recognitioncited by 5
  25. 2022Avoid Overthinking in Self-Supervised Models for Speech Recognitioncited by 5
  26. 2023A CTC Alignment-based Non-autoregressive Transformer for End-to-end Automatic Speech Recognitioncited by 5
  27. 2023Semi-Autoregressive Streaming ASR With Label Contextcited by 5
  28. 2024CNVSRC 2023: The First Chinese Continuous Visual Speech Recognition Challengecited by 5
  29. 2024InterBiasing: Boost Unseen Word Recognition through Biasing Intermediate Predictionscited by 5
  30. 2026An Empirical Recipe for Universal Phone Recognitioncited by 5
  31. 2022Improving CTC-based ASR Models with Gated Interlayer Collaborationcited by 4
  32. 2023A New Benchmark of Aphasia Speech Recognition and Detection Based on E-Branchformer and Multi-task Learningcited by 4
  33. 2023SSHR: Leveraging Self-supervised Hierarchical Representations for Multilingual Automatic Speech Recognitioncited by 4
  34. 2023Soft Alignment of Modality Space for End-to-end Speech Translationcited by 4
  35. 2025Accelerating Diffusion-based Text-to-Speech Model Training with Dual Modality Alignmentcited by 4
  36. 2026PRiSM: Benchmarking Phone Realization in Speech Modelscited by 4
  37. 2021PM-MMUT: Boosted Phone-Mask Data Augmentation using Multi-Modeling Unit Training for Phonetic-Reduction-Robust E2E Speech Recognitioncited by 3
  38. 2022MLP-ASR: Sequence-length agnostic all-MLP architectures for speech recognitioncited by 3

50 most cited of 95 All 95 (JSON)