mlsanityData to 7 Oct 2026

Multiple Representation Transfer from Large Language Models to End-to-End ASR Systems

arXiv 2309.04031 ↗First posted 7 Sep 2023cs.CL cs.SD eess.AScited by 2

Not on the map. Why?

Abstract (arXiv)

Transferring the knowledge of large language models (LLMs) is a promising technique to incorporate linguistic knowledge into end-to-end automatic speech recognition (ASR) systems. However, existing works only transfer a single representation of LLM (e.g. the last layer of pretrained BERT), while the representation of a text is inherently non-unique and can be obtained variously from different layers, contexts and models. In this work, we explore a wide range of techniques to obtain and transfer multiple representations of LLMs into a transducer-based ASR system. While being conceptually simple, we show that transferring multiple representations of LLMs can be an effective alternative to transferring only a single representation.

Key facts. Cited by 2 papers in the map (data through 7 Oct 2026). Builds on 35 of its 41 references that are in the map. First posted on arXiv 7 Sep 2023.

Builds on

35 of 41 references.

References · 35

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2020Language Models are Few-Shot Learnerscited by 32,199
  4. 2023GPT-4 Technical Reportcited by 18,132
  5. 2022Training language models to follow instructions with human feedbackcited by 15,712
  6. 2015Distilling the Knowledge in a Neural Networkcited by 10,076
  7. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,854
  8. 2019DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lightercited by 4,243
  9. 2019Unsupervised Cross-lingual Representation Learning at Scalecited by 3,907
  10. 2020Conformer: Convolution-augmented Transformer for Speech Recognitioncited by 1,838
  11. 2019TinyBERT: Distilling BERT for Natural Language Understandingcited by 1,153
  12. 2019BERT Rediscovers the Classical NLP Pipelinecited by 1,046
23 more
  1. 2020XTREME: A Massively Multilingual Multi-task Benchmark for Evaluating Cross-lingual Generalizationcited by 738
  2. 2019Patient Knowledge Distillation for BERT Model Compressioncited by 502
  3. 2023AudioPaLM: A Large Language Model That Can Speak and Listencited by 333
  4. 2020A Density Ratio Approach to Language Model Fusion in End-To-End Automatic Speech Recognitioncited by 73
  5. 2018A Comparison of Techniques for Language Model Integration in Encoder-Decoder Speech Recognitioncited by 66
  6. 2021Advancing RNN Transducer Technology for Speech Recognitioncited by 44
  7. 2020Distilling the Knowledge of BERT for Sequence-to-Sequence ASRcited by 38
  8. 2021Fast End-to-End Speech Recognition via Non-Autoregressive Models and Cross-Modal Knowledge Transferring from BERTcited by 31
  9. 2021Innovative Bert-based Reranking Language Models for Speech Recognitioncited by 23
  10. 2022Effect and Analysis of Large-scale Language Model Rescoring on Competitive ASR Systemscited by 23
  11. 2021BART based semantic correction for Mandarin automatic speech recognition systemcited by 20
  12. 2022Knowledge Transfer from Large-scale Pretrained Language Models to End-to-end Speech Recognizerscited by 19
  13. 2021RAIL-KD: RAndom Intermediate Layer Mapping for Knowledge Distillationcited by 18
  14. 2021Efficiently Fusing Pretrained Acoustic and Linguistic Encoders for Low-resource Speech Recognitioncited by 17
  15. 2021Wav-BERT: Cooperative Acoustic and Linguistic Representation Learning for Low-Resource Speech Recognitioncited by 16
  16. 2022Improving CTC-based speech recognition via knowledge transferring from pre-trained language modelscited by 14
  17. 2022A context-aware knowledge transferring strategy for CTC-based ASRcited by 12
  18. 2021Speech Recognition by Simply Fine-tuning BERTcited by 11
  19. 2022BERT Meets CTC: New Formulation of End-to-End Speech Recognition with Pre-trained Masked Language Modelcited by 11
  20. 2023Knowledge Transfer from Pre-trained Language Models to Cif-based Speech Recognizers via Hierarchical Distillationcited by 11
  21. 2022BECTRA: Transducer-based End-to-End ASR with BERT-Enhanced Encodercited by 8
  22. 2023Exploring the Integration of Large Language Models into Automatic Speech Recognition Systems: An Empirical Studycited by 7
  23. 2019Sequence-to-sequence Automatic Speech Recognition with Word Embedding Regularization and Fused Decodingcited by 3

Built on by

Cited by year

Citing papers per year: 2024: 2.

2 papers.

  1. 2024The Sound of Healthcare: Improving Medical Transcription ASR Accuracy with Large Language Modelscited by 17
  2. 2024Mamba-based Decoder-Only Approach with Bidirectional Speech Modeling for Speech Recognitioncited by 5