mlsanityData to 14 Oct 2026

Low-rank Adaptation of Large Language Model Rescoring for Parameter-Efficient Speech Recognition

arXiv 2309.15223 ↗First posted 26 Sep 2023cs.CL cs.AI cs.LGcited by 13

Not on the map. Why?

Abstract (arXiv)

We propose a neural language modeling system based on low-rank adaptation (LoRA) for speech recognition output rescoring. Although pretrained language models (LMs) like BERT have shown superior performance in second-pass rescoring, the high computational cost of scaling up the pretraining stage and adapting the pretrained models to specific domains limit their practical use in rescoring. Here we present a method based on low-rank decomposition to train a rescoring BERT model and adapt it to new domains using only a fraction (0.08%) of the pretrained parameters. These inserted matrices are optimized through a discriminative training objective along with a correlation-based regularization loss. The proposed low-rank adaptation Rescore-BERT (LoRB) architecture is evaluated on LibriSpeech and internal datasets with decreased training times by factors between 5.4 and 3.6.

Key facts. Cited by 13 papers in the map (data through 14 Oct 2026). Builds on 41 of its 51 references that are in the map. First posted on arXiv 26 Sep 2023.

Builds on

41 of 51 references.

References · 41

  1. 2017Attention Is All You Needcited by 60,361
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  3. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
  4. 2019Parameter-Efficient Transfer Learning for NLPcited by 4,236
  5. 2021Prefix-Tuning: Optimizing Continuous Prompts for Generationcited by 3,915
  6. 2021BitFit: Simple Parameter-efficient Fine-tuning for Transformer-based Masked Language-modelscited by 1,245
  7. 2020Intrinsic Dimensionality Explains the Effectiveness of Language Model Fine-Tuningcited by 671
  8. 2019Representation Degeneration Problem in Training Natural Language Generation Modelscited by 255
  9. 2019SMART: Robust and Efficient Fine-Tuning for Pre-trained Natural Language Models through Principled Regularized Optimizationcited by 254
  10. 2022DyLoRA: Parameter Efficient Tuning of Pre-trained Models using Dynamic Search-Free Low-Rank Adaptationcited by 233
  11. 2020Better Fine-Tuning by Reducing Representational Collapsecited by 157
  12. 2022On the Effectiveness of Parameter-Efficient Fine-Tuningcited by 141
29 more
  1. 2021Voice2Series: Reprogramming Acoustic Models for Time Series Classificationcited by 91
  2. 2023One-for-All: Generalized LoRA for Parameter-Efficient Fine-tuningcited by 85
  3. 2017Minimum Word Error Rate Training for Attention-based Sequence-to-Sequence Modelscited by 68
  4. 2019Two-Pass End-to-End Speech Recognitioncited by 64
  5. 2023Empirical Analysis of the Strengths and Weaknesses of PEFT Techniques for LLMscited by 38
  6. 2022RescoreBERT: Discriminative Speech Recognition Rescoring with BERTcited by 34
  7. 2023SpeechPrompt v2: Prompt Tuning for Speech Classification Taskscited by 32
  8. 2021Transformer Based Deliberation for Two-Pass Speech Recognitioncited by 29
  9. 2021Large-scale ASR Domain Adaptation using Self- and Semi-supervised Learningcited by 26
  10. 2022Exploring Efficient-tuning Methods in Self-supervised Speech Modelscited by 26
  11. 2023Neural Architecture Search for Parameter-Efficient Fine-tuning of Large Pre-trained Language Modelscited by 25
  12. 2023From English to More Languages: Parameter-Efficient Model Reprogramming for Cross-Lingual Speech Recognitioncited by 23
  13. 2020Multi-task Language Modeling for Improving Speech Recognition of Rare Wordscited by 21
  14. 2022SpeechPrompt: An Exploration of Prompt Tuning on Generative Spoken Language Model for Speech Processing Taskscited by 19
  15. 2022Fine-tuning Pre-trained Language Models for Few-shot Intent Detection: Supervised Pre-training and Isotropizationcited by 19
  16. 2022Low-Resource Music Genre Classification with Cross-Modal Neural Model Reprogrammingcited by 17
  17. 2021Adapting GPT, GPT-2 and BERT Language Models for Speech Recognitioncited by 16
  18. 2019Audio-attention discriminative language model for ASR rescoringcited by 13
  19. 2021Domain-aware Neural Language Models for Speech Recognitioncited by 13
  20. 2022CHAPTER: Exploiting Convolutional Neural Network Adapters for Self-supervised Speech Modelscited by 13
  21. 2022Motion Style Transfer: Modular Low-Rank Adaptation for Deep Motion Forecastingcited by 12
  22. 2023PROCTER: PROnunciation-aware ConTextual adaptER for personalized speech recognition in neural transducerscited by 10
  23. 2023How to Estimate Model Transferability of Pre-Trained Speech Models?cited by 10
  24. 2022A Likelihood Ratio based Domain Adaptation Method for E2E Modelscited by 5
  25. 2023Scaling Laws for Discriminative Speech Recognition Rescoring Modelscited by 5
  26. 2023Parameter-Efficient Learning for Text-to-Speech Accent Adaptationcited by 4
  27. 2022Evaluating Parameter Efficient Learning for Generationcited by 3
  28. 2023Distillation Strategies for Discriminative Speech Recognition Rescoringcited by 2
  29. 2023Differentially Private Adapters for Parameter Efficient Acoustic Modelingcited by 1

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 1; 2024: 6; 2025: 5; 2026: 1.

13 papers.

Sort:

By area:

  1. 2023Whispering LLaMA: A Cross-Modal Generative Error Correction Framework for Speech Recognitioncited by 56
  2. 2024GenTranslate: Large Language Models are Generative Multilingual Speech and Machine Translatorscited by 22
  3. 2024Balancing Continuous Pre-Training and Instruction Fine-Tuning: Optimizing Instruction-Following in LLMscited by 12
  4. 2025Don't Trust Generative Agents to Mimic Communication on Social Networks Unless You Benchmarked their Empirical Realismcited by 6
  5. 2024Let's Fuse Step by Step: A Generative Fusion Decoding Algorithm with LLMs for Robust and Instruction-Aware ASR and OCRcited by 4
  6. 2025Less Data, More Security: Advancing Cybersecurity LLMs Specialization via Resource-Efficient Domain-Adaptive Continuous Pre-training with Minimal Tokenscited by 4
  7. 2024Backdoor in Seconds: Unlocking Vulnerabilities in Large Pre-trained Models via Model Editingcited by 3
  8. 2025Context Parametrization with Compositional Adapterscited by 3
  9. 2025Ref-Long: Benchmarking the Long-context Referencing Capability of Long-context Language Modelscited by 2
  10. 2024Leveraging Parameter-Efficient Transfer Learning for Multi-Lingual Text-to-Speech Adaptationcited by 1
  11. 2024Residual Multi-Task Learner for Applied Rankingcited by 1
  12. 2025f-INE: A Hypothesis Testing Framework for Estimating Influence under Training Randomnesscited by 0
1 more
  1. 2026Generative Chemical Language Models for Energetic Materials Discoverycited by 0