# A context-aware knowledge transferring strategy for CTC-based ASR arXiv: https://arxiv.org/abs/2210.06244 ยท HTML: https://mlsanity.com/paper/2210.06244/ ## Key facts - Cited by 12 papers in the map (data through 14 Oct 2026). - Builds on 31 of its 38 references that are in the map. - First posted on arXiv 12 Oct 2022. ## Builds on (most cited first, 30 of 31) - [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017) - [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018) - [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019) - [wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representations](https://mlsanity.com/paper/2006.11477/) (2020) - [XLNet: Generalized Autoregressive Pretraining for Language Understanding](https://mlsanity.com/paper/1906.08237/) (2019) - [HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Units](https://mlsanity.com/paper/2106.07447/) (2021) - [fairseq: A Fast, Extensible Toolkit for Sequence Modeling](https://mlsanity.com/paper/1904.01038/) (2019) - [ESPnet: End-to-End Speech Processing Toolkit](https://mlsanity.com/paper/1804.00015/) (2018) - [AISHELL-1: An Open-Source Mandarin Speech Corpus and A Speech Recognition Baseline](https://mlsanity.com/paper/1709.05522/) (2017) - [vq-wav2vec: Self-Supervised Learning of Discrete Speech Representations](https://mlsanity.com/paper/1910.05453/) (2019) - [AISHELL-2: Transforming Mandarin ASR Research Into Industrial Scale](https://mlsanity.com/paper/1808.10583/) (2018) - [Intermediate Loss Regularization for CTC-based Speech Recognition](https://mlsanity.com/paper/2102.03216/) (2021) - [How Does BERT Answer Questions? A Layer-Wise Analysis of Transformer Representations](https://mlsanity.com/paper/1909.04925/) (2019) - [Relaxing the Conditional Independence Assumption of CTC-based ASR by Conditioning on Intermediate Predictions](https://mlsanity.com/paper/2104.02724/) (2021) - [Attention Flows: Analyzing and Comparing Attention Mechanisms in Language Models](https://mlsanity.com/paper/2009.07053/) (2020) - [Distilling the Knowledge of BERT for Sequence-to-Sequence ASR](https://mlsanity.com/paper/2008.03822/) (2020) - [Incorporating BERT into Parallel Sequence Decoding with Adapters](https://mlsanity.com/paper/2010.06138/) (2020) - [What's so special about BERT's layers? A closer look at the NLP pipeline in monolingual and multilingual models](https://mlsanity.com/paper/2004.06499/) (2020) - [Fast End-to-End Speech Recognition via Non-Autoregressive Models and Cross-Modal Knowledge Transferring from BERT](https://mlsanity.com/paper/2102.07594/) (2021) - [A Comparative Study on Non-Autoregressive Modelings for Speech-to-Text Generation](https://mlsanity.com/paper/2110.05249/) (2021) - [Non-Autoregressive Transformer ASR with CTC-Enhanced Decoder Input](https://mlsanity.com/paper/2010.15025/) (2020) - [Improving non-autoregressive end-to-end speech recognition with pre-trained acoustic and language models](https://mlsanity.com/paper/2201.10103/) (2022) - [Knowledge Transfer from Large-scale Pretrained Language Models to End-to-end Speech Recognizers](https://mlsanity.com/paper/2202.07894/) (2022) - [Efficiently Fusing Pretrained Acoustic and Linguistic Encoders for Low-resource Speech Recognition](https://mlsanity.com/paper/2101.06699/) (2021) - [Wav-BERT: Cooperative Acoustic and Linguistic Representation Learning for Low-Resource Speech Recognition](https://mlsanity.com/paper/2109.09161/) (2021) - [Improving Hybrid CTC/Attention End-to-end Speech Recognition with Pretrained Acoustic and Language Model](https://mlsanity.com/paper/2112.07254/) (2021) - [Improving CTC-based speech recognition via knowledge transferring from pre-trained language models](https://mlsanity.com/paper/2203.03582/) (2022) - [Non-autoregressive Transformer-based End-to-end ASR using BERT](https://mlsanity.com/paper/2104.04805/) (2021) - [Hierarchical Conditional End-to-End ASR with CTC and Multi-Granular Subword Units](https://mlsanity.com/paper/2110.04109/) (2021) - [Speech Recognition by Simply Fine-tuning BERT](https://mlsanity.com/paper/2102.00291/) (2021) ## Built on by (most cited first, 12 of 12) - [Encoding of lexical tone in self-supervised models of spoken language](https://mlsanity.com/paper/2403.16865/) (2024) - [A layer-wise analysis of Mandarin and English suprasegmentals in SSL speech models](https://mlsanity.com/paper/2408.13678/) (2024) - [BECTRA: Transducer-based End-to-End ASR with BERT-Enhanced Encoder](https://mlsanity.com/paper/2211.00792/) (2022) - [Temporal Order Preserved Optimal Transport-based Cross-modal Knowledge Transfer Learning for ASR](https://mlsanity.com/paper/2409.02239/) (2024) - [Analyzing the relationships between pretraining language, phonetic, tonal, and speaker information in self-supervised speech models](https://mlsanity.com/paper/2506.10855/) (2025) - [Multiple Representation Transfer from Large Language Models to End-to-End ASR Systems](https://mlsanity.com/paper/2309.04031/) (2023) - [HypR: A comprehensive study for ASR hypothesis revising with a reference corpus](https://mlsanity.com/paper/2309.09838/) (2023) - [Cross-modal Knowledge Transfer Learning as Graph Matching Based on Optimal Transport for ASR](https://mlsanity.com/paper/2505.13079/) (2025) - [LTA-L2S: Lexical Tone-Aware Lip-to-Speech Synthesis for Mandarin with Cross-Lingual Transfer Learning](https://mlsanity.com/paper/2509.25670/) (2025) - [A Lexical-aware Non-autoregressive Transformer-based ASR Model](https://mlsanity.com/paper/2305.10839/) (2023) - [New Insights into Optimal Alignment of Acoustic and Linguistic Representations for Knowledge Transfer in ASR](https://mlsanity.com/paper/2509.05609/) (2025) - [Optimal transport meets speech: a tutorial review](https://mlsanity.com/paper/2609.31787/) (2026) ## Abstract (arXiv) Non-autoregressive automatic speech recognition (ASR) modeling has received increasing attention recently because of its fast decoding speed and superior performance. Among representatives, methods based on the connectionist temporal classification (CTC) are still a dominating stream. However, the theoretically inherent flaw, the assumption of independence between tokens, creates a performance barrier for the school of works. To mitigate the challenge, we propose a context-aware knowledge transferring strategy, consisting of a knowledge transferring module and a context-aware training strategy, for CTC-based ASR. The former is designed to distill linguistic information from a pre-trained language model, and the latter is framed to modulate the limitations caused by the conditional independence assumption. As a result, a knowledge-injected context-aware CTC-based ASR built upon the wav2vec2.0 is presented in this paper. A series of experiments on the AISHELL-1 and AISHELL-2 datasets demonstrate the effectiveness of the proposed method.