mlsanityData to 7 Oct 2026

Distilling the Knowledge of BERT for Sequence-to-Sequence ASR

arXiv 2008.03822 ↗First posted 9 Aug 2020cs.CL eess.AScited by 38

Not on the map. Why?

Abstract (arXiv)

Attention-based sequence-to-sequence (seq2seq) models have achieved promising results in automatic speech recognition (ASR). However, as these models decode in a left-to-right way, they do not have access to context on the right. We leverage both left and right context by applying BERT as an external language model to seq2seq ASR through knowledge distillation. In our proposed method, BERT generates soft labels to guide the training of seq2seq ASR. Furthermore, we leverage context beyond the current utterance as input to BERT. Experimental evaluations show that our method significantly improves the ASR performance from the seq2seq baseline on the Corpus of Spontaneous Japanese (CSJ). Knowledge distillation from BERT outperforms that from a transformer LM that only looks at left context. We also show the effectiveness of leveraging context beyond the current utterance. Our method outperforms other LM application approaches such as n-best rescoring and shallow fusion, while it does not require extra inference cost.

Key facts. Cited by 38 papers in the map (data through 7 Oct 2026). First posted on arXiv 9 Aug 2020.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 12; 2022: 10; 2023: 6; 2024: 5; 2025: 3; 2026: 2.

38 papers.

Sort:

By area:

  1. 2021Recent Advances in End-to-End Automatic Speech Recognitioncited by 138
  2. 2021Fast End-to-End Speech Recognition via Non-Autoregressive Models and Cross-Modal Knowledge Transferring from BERTcited by 31
  3. 2022Effect and Analysis of Large-scale Language Model Rescoring on Competitive ASR Systemscited by 23
  4. 2024It's Never Too Late: Fusing Acoustic Information into Large Language Models for Automatic Speech Recognitioncited by 22
  5. 2022Knowledge Transfer from Large-scale Pretrained Language Models to End-to-end Speech Recognizerscited by 19
  6. 2021Shrinking Bigfoot: Reducing wav2vec 2.0 footprintcited by 16
  7. 2021Wav-BERT: Cooperative Acoustic and Linguistic Representation Learning for Low-Resource Speech Recognitioncited by 16
  8. 2021Knowledge Distillation from BERT Transformer to Speech Transformer for Intent Classificationcited by 14
  9. 2021Improving Hybrid CTC/Attention End-to-end Speech Recognition with Pretrained Acoustic and Language Modelcited by 14
  10. 2022Improving CTC-based speech recognition via knowledge transferring from pre-trained language modelscited by 14
  11. 2021Non-autoregressive Transformer-based End-to-end ASR using BERTcited by 13
  12. 2021Temporal Knowledge Distillation for On-device Audio Classificationcited by 13
26 more
  1. 2021Hierarchical Transformer-based Large-Context End-to-end ASR with Large-Context Knowledge Distillationcited by 12
  2. 2022A context-aware knowledge transferring strategy for CTC-based ASRcited by 12
  3. 2021Speech Recognition by Simply Fine-tuning BERTcited by 11
  4. 2022BERT Meets CTC: New Formulation of End-to-End Speech Recognition with Pre-trained Masked Language Modelcited by 11
  5. 2023Knowledge Transfer from Pre-trained Language Models to Cif-based Speech Recognizers via Hierarchical Distillationcited by 11
  6. 2022Simple Pooling Front-ends For Efficient Audio Classificationcited by 8
  7. 2022BECTRA: Transducer-based End-to-End ASR with BERT-Enhanced Encodercited by 8
  8. 2024Automatic Speech Recognition with BERT and CTC Transformers: A Reviewcited by 8
  9. 2023Exploring the Integration of Large Language Models into Automatic Speech Recognition Systems: An Empirical Studycited by 7
  10. 2022Distilling the Knowledge of BERT for CTC-based ASRcited by 6
  11. 2022Non-autoregressive Error Correction for CTC-based ASR with Phone-conditioned Masked LMcited by 6
  12. 2023LOST: A Mental Health Dataset of Low Self-esteem in Reddit Postscited by 6
  13. 2024Benchmarking Japanese Speech Recognition on ASR-LLM Setups with Multi-Pass Augmented Generative Error Correctioncited by 4
  14. 2022Improving Automatic Speech Recognition for Non-Native English with Transfer Learning and Language Model Decodingcited by 3
  15. 2024Keep Decoding Parallel with Effective Knowledge Distillation from Language Models to End-to-end Speech Recogniserscited by 3
  16. 2025Linguistic Knowledge Transfer Learning for Speech Enhancementcited by 3
  17. 2025LLM-based Generative Error Correction for Rare Words with Synthetic Data and Phonetic Contextcited by 3
  18. 2023Multiple Representation Transfer from Large Language Models to End-to-End ASR Systemscited by 2
  19. 2021Global Structure-Aware Drum Transcription Based on Self-Attention Mechanismscited by 1
  20. 2021Knowledge distillation from language model to acoustic model: a hierarchical multi-task learning approachcited by 1
  21. 2023Topics in Contextualised Attention Embeddingscited by 1
  22. 2023VILAS: Exploring the Effects of Vision and Language Context in Automatic Speech Recognitioncited by 1
  23. 2024Empowering the Deaf and Hard of Hearing Community: Enhancing Video Captions Using Large Language Modelscited by 0
  24. 2025MTLM: Incorporating Bidirectional Text Information to Enhance Language Model Training in Speech Recognition Systemscited by 0
  25. 2026Listen, Attend, Understand: a Regularization Technique for Stable E2E Speech Translation Training on High Variance labelscited by 0
  26. 2026Merging the Knowledge of LLMs for Automatic Speech Recognitioncited by 0