mlsanityData to 7 Oct 2026

A context-aware knowledge transferring strategy for CTC-based ASR

arXiv 2210.06244 ↗First posted 12 Oct 2022cs.CL cs.SD eess.AScited by 12

Not on the map. Why?

Abstract (arXiv)

Non-autoregressive automatic speech recognition (ASR) modeling has received increasing attention recently because of its fast decoding speed and superior performance. Among representatives, methods based on the connectionist temporal classification (CTC) are still a dominating stream. However, the theoretically inherent flaw, the assumption of independence between tokens, creates a performance barrier for the school of works. To mitigate the challenge, we propose a context-aware knowledge transferring strategy, consisting of a knowledge transferring module and a context-aware training strategy, for CTC-based ASR. The former is designed to distill linguistic information from a pre-trained language model, and the latter is framed to modulate the limitations caused by the conditional independence assumption. As a result, a knowledge-injected context-aware CTC-based ASR built upon the wav2vec2.0 is presented in this paper. A series of experiments on the AISHELL-1 and AISHELL-2 datasets demonstrate the effectiveness of the proposed method.

Key facts. Cited by 12 papers in the map (data through 7 Oct 2026). Builds on 31 of its 38 references that are in the map. First posted on arXiv 12 Oct 2022.

Builds on

31 of 38 references.

References · 31

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  4. 2020wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representationscited by 3,853
  5. 2019XLNet: Generalized Autoregressive Pretraining for Language Understandingcited by 2,801
  6. 2021HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Unitscited by 2,301
  7. 2019fairseq: A Fast, Extensible Toolkit for Sequence Modelingcited by 1,486
  8. 2018ESPnet: End-to-End Speech Processing Toolkitcited by 554
  9. 2017AISHELL-1: An Open-Source Mandarin Speech Corpus and A Speech Recognition Baselinecited by 364
  10. 2019vq-wav2vec: Self-Supervised Learning of Discrete Speech Representationscited by 335
  11. 2018AISHELL-2: Transforming Mandarin ASR Research Into Industrial Scalecited by 125
  12. 2021Intermediate Loss Regularization for CTC-based Speech Recognitioncited by 95
19 more
  1. 2019How Does BERT Answer Questions? A Layer-Wise Analysis of Transformer Representationscited by 63
  2. 2021Relaxing the Conditional Independence Assumption of CTC-based ASR by Conditioning on Intermediate Predictionscited by 61
  3. 2020Attention Flows: Analyzing and Comparing Attention Mechanisms in Language Modelscited by 49
  4. 2020Incorporating BERT into Parallel Sequence Decoding with Adapterscited by 39
  5. 2020Distilling the Knowledge of BERT for Sequence-to-Sequence ASRcited by 38
  6. 2020What's so special about BERT's layers? A closer look at the NLP pipeline in monolingual and multilingual modelscited by 35
  7. 2021Fast End-to-End Speech Recognition via Non-Autoregressive Models and Cross-Modal Knowledge Transferring from BERTcited by 31
  8. 2021A Comparative Study on Non-Autoregressive Modelings for Speech-to-Text Generationcited by 27
  9. 2020Non-Autoregressive Transformer ASR with CTC-Enhanced Decoder Inputcited by 24
  10. 2022Improving non-autoregressive end-to-end speech recognition with pre-trained acoustic and language modelscited by 21
  11. 2022Knowledge Transfer from Large-scale Pretrained Language Models to End-to-end Speech Recognizerscited by 19
  12. 2021Efficiently Fusing Pretrained Acoustic and Linguistic Encoders for Low-resource Speech Recognitioncited by 17
  13. 2021Wav-BERT: Cooperative Acoustic and Linguistic Representation Learning for Low-Resource Speech Recognitioncited by 16
  14. 2021Improving Hybrid CTC/Attention End-to-end Speech Recognition with Pretrained Acoustic and Language Modelcited by 14
  15. 2022Improving CTC-based speech recognition via knowledge transferring from pre-trained language modelscited by 14
  16. 2021Non-autoregressive Transformer-based End-to-end ASR using BERTcited by 13
  17. 2021Hierarchical Conditional End-to-End ASR with CTC and Multi-Granular Subword Unitscited by 13
  18. 2021Speech Recognition by Simply Fine-tuning BERTcited by 11
  19. 2020Reducing Spelling Inconsistencies in Code-Switching ASR using Contextualized CTC Losscited by 4

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 1; 2023: 3; 2024: 3; 2025: 4; 2026: 1.

12 papers.

  1. 2024Encoding of lexical tone in self-supervised models of spoken languagecited by 22
  2. 2024A layer-wise analysis of Mandarin and English suprasegmentals in SSL speech modelscited by 18
  3. 2022BECTRA: Transducer-based End-to-End ASR with BERT-Enhanced Encodercited by 8
  4. 2024Temporal Order Preserved Optimal Transport-based Cross-modal Knowledge Transfer Learning for ASRcited by 3
  5. 2025Analyzing the relationships between pretraining language, phonetic, tonal, and speaker information in self-supervised speech modelscited by 3
  6. 2023Multiple Representation Transfer from Large Language Models to End-to-End ASR Systemscited by 2
  7. 2023HypR: A comprehensive study for ASR hypothesis revising with a reference corpuscited by 1
  8. 2025Cross-modal Knowledge Transfer Learning as Graph Matching Based on Optimal Transport for ASRcited by 1
  9. 2025LTA-L2S: Lexical Tone-Aware Lip-to-Speech Synthesis for Mandarin with Cross-Lingual Transfer Learningcited by 1
  10. 2023A Lexical-aware Non-autoregressive Transformer-based ASR Modelcited by 0
  11. 2025New Insights into Optimal Alignment of Acoustic and Linguistic Representations for Knowledge Transfer in ASRcited by 0
  12. 2026Optimal transport meets speech: a tutorial reviewcited by 0