mlsanityData to 7 Oct 2026

Knowledge Transfer from Pre-trained Language Models to Cif-based Speech Recognizers via Hierarchical Distillation

arXiv 2301.13003 ↗First posted 30 Jan 2023cs.CL cs.AI cs.SDcited by 11

Not on the map. Why?

Abstract (arXiv)

Large-scale pre-trained language models (PLMs) have shown great potential in natural language processing tasks. Leveraging the capabilities of PLMs to enhance automatic speech recognition (ASR) systems has also emerged as a promising research direction. However, previous works may be limited by the inflexible structures of PLMs and the insufficient utilization of PLMs. To alleviate these problems, we propose the hierarchical knowledge distillation (HKD) on the continuous integrate-and-fire (CIF) based ASR models. To transfer knowledge from PLMs to the ASR models, HKD employs cross-modal knowledge distillation with contrastive loss at the acoustic level and knowledge distillation with regression loss at the linguistic level. Compared with the original CIF-based model, our method achieves 15% and 9% relative error rate reduction on the AISHELL-1 and LibriSpeech datasets, respectively.

Key facts. Cited by 11 papers in the map (data through 7 Oct 2026). Builds on 36 of its 45 references that are in the map. First posted on arXiv 30 Jan 2023.

Builds on

36 of 45 references.

References · 36

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  4. 2015Distilling the Knowledge in a Neural Networkcited by 10,076
  5. 2018Representation Learning with Contrastive Predictive Codingcited by 7,662
  6. 2020Conformer: Convolution-augmented Transformer for Speech Recognitioncited by 1,838
  7. 2019SpecAugment: A Simple Data Augmentation Method for Automatic Speech Recognitioncited by 1,355
  8. 2019Contrastive Representation Distillationcited by 734
  9. 2018ESPnet: End-to-End Speech Processing Toolkitcited by 554
  10. 2015Listen, Attend and Spellcited by 542
  11. 2017AISHELL-1: An Open-Source Mandarin Speech Corpus and A Speech Recognition Baselinecited by 364
  12. 2015On Using Monolingual Corpora in Neural Machine Translationcited by 149
24 more
  1. 2022Paraformer: Fast and Accurate Parallel Transformer for Non-autoregressive End-to-End Speech Recognitioncited by 139
  2. 2021WeNet: Production oriented Streaming and Non-streaming End-to-End Speech Recognition Toolkitcited by 123
  3. 2022Branchformer: Parallel MLP-Attention Architectures to Capture Local and Global Context for Speech Recognition and Understandingcited by 90
  4. 2017Cold Fusion: Training Seq2Seq Models Together with Language Modelscited by 87
  5. 2020Internal Language Model Estimation for Domain-Adaptive End-to-End Speech Recognitioncited by 80
  6. 2019CIF: Continuous Integrate-and-Fire for End-to-End Speech Recognitioncited by 76
  7. 2018A Comparison of Techniques for Language Model Integration in Encoder-Decoder Speech Recognitioncited by 66
  8. 2022Improving End-to-End Contextual Speech Recognition with Fine-Grained Contextual Knowledge Selectioncited by 39
  9. 2020Distilling the Knowledge of BERT for Sequence-to-Sequence ASRcited by 38
  10. 2022RescoreBERT: Discriminative Speech Recognition Rescoring with BERTcited by 34
  11. 2019Self-Attention Aligner: A Latency-Control End-to-End Model for ASR Using Self-Attention Network and Chunk-Hoppingcited by 33
  12. 2021Fast End-to-End Speech Recognition via Non-Autoregressive Models and Cross-Modal Knowledge Transferring from BERTcited by 31
  13. 2020LRC-BERT: Latent-representation Contrastive Knowledge Distillation for Natural Language Understandingcited by 27
  14. 2021Innovative Bert-based Reranking Language Models for Speech Recognitioncited by 23
  15. 2022Improving non-autoregressive end-to-end speech recognition with pre-trained acoustic and language modelscited by 21
  16. 2020CIF-based Collaborative Decoding for End-to-end Contextual Speech Recognitioncited by 20
  17. 2022Knowledge Transfer from Large-scale Pretrained Language Models to End-to-end Speech Recognizerscited by 19
  18. 2021Efficiently Fusing Pretrained Acoustic and Linguistic Encoders for Low-resource Speech Recognitioncited by 17
  19. 2021Adapting GPT, GPT-2 and BERT Language Models for Speech Recognitioncited by 16
  20. 2021Wav-BERT: Cooperative Acoustic and Linguistic Representation Learning for Low-Resource Speech Recognitioncited by 16
  21. 2021ASR Rescoring and Confidence Estimation with ELECTRAcited by 16
  22. 2022Improving CTC-based speech recognition via knowledge transferring from pre-trained language modelscited by 14
  23. 2021Speech Recognition by Simply Fine-tuning BERTcited by 11
  24. 2022Integrating Lattice-Free MMI into End-to-End Speech Recognitioncited by 3

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 3; 2024: 2; 2025: 5; 2026: 1.

11 papers.

  1. 2024MM-LLMs: Recent Advances in MultiModal Large Language Modelscited by 277
  2. 2024Temporal Order Preserved Optimal Transport-based Cross-modal Knowledge Transfer Learning for ASRcited by 3
  3. 2023Multiple Representation Transfer from Large Language Models to End-to-End ASR Systemscited by 2
  4. 2023Correction Focused Language Model Training for Speech Recognitioncited by 2
  5. 2023VILAS: Exploring the Effects of Vision and Language Context in Automatic Speech Recognitioncited by 1
  6. 2025Cross-modal Knowledge Transfer Learning as Graph Matching Based on Optimal Transport for ASRcited by 1
  7. 2025Incorporating Linguistic Constraints from External Knowledge Source for Audio-Visual Target Speech Extractioncited by 1
  8. 2025DESign: Dynamic Context-Aware Convolution and Efficient Subnet Regularization for Continuous Sign Language Recognitioncited by 0
  9. 2025New Insights into Optimal Alignment of Acoustic and Linguistic Representations for Knowledge Transfer in ASRcited by 0
  10. 2025M-CIF: Multi-Scale Alignment For CIF-Based Non-Autoregressive ASRcited by 0
  11. 2026Optimal transport meets speech: a tutorial reviewcited by 0