# Effect and Analysis of Large-scale Language Model Rescoring on Competitive ASR Systems

arXiv: https://arxiv.org/abs/2204.00212 · HTML: https://mlsanity.com/paper/2204.00212/

## Key facts

- Cited by 23 papers in the map (data through 14 Oct 2026).
- Builds on 20 of its 34 references that are in the map.
- First posted on arXiv 1 Apr 2022.

## Builds on (most cited first, 20 of 20)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [Language Models as Knowledge Bases?](https://mlsanity.com/paper/1909.01066/) (2019)
- [Conformer: Convolution-augmented Transformer for Speech Recognition](https://mlsanity.com/paper/2005.08100/) (2020)
- [SpecAugment: A Simple Data Augmentation Method for Automatic Speech Recognition](https://mlsanity.com/paper/1904.08779/) (2019)
- [BERT Rediscovers the Classical NLP Pipeline](https://mlsanity.com/paper/1905.05950/) (2019)
- [BLiMP: The Benchmark of Linguistic Minimal Pairs for English](https://mlsanity.com/paper/1912.00582/) (2019)
- [What do you learn from context? Probing for sentence structure in contextualized word representations](https://mlsanity.com/paper/1905.06316/) (2019)
- [BERT has a Mouth, and It Must Speak: BERT as a Markov Random Field Language Model](https://mlsanity.com/paper/1902.04094/) (2019)
- [Advancing RNN Transducer Technology for Speech Recognition](https://mlsanity.com/paper/2103.09935/) (2021)
- [Distilling the Knowledge of BERT for Sequence-to-Sequence ASR](https://mlsanity.com/paper/2008.03822/) (2020)
- [English Conversational Telephone Speech Recognition by Humans and Machines](https://mlsanity.com/paper/1703.02136/) (2017)
- [Single headed attention based sequence-to-sequence model for state-of-the-art results on Switchboard](https://mlsanity.com/paper/2001.07263/) (2020)
- [RescoreBERT: Discriminative Speech Recognition Rescoring with BERT](https://mlsanity.com/paper/2202.01094/) (2022)
- [Fast and Accurate Recurrent Neural Network Acoustic Models for Speech Recognition](https://mlsanity.com/paper/1507.06947/) (2015)
- [Innovative Bert-based Reranking Language Models for Speech Recognition](https://mlsanity.com/paper/2104.04950/) (2021)
- [Knowledge Transfer from Large-scale Pretrained Language Models to End-to-end Speech Recognizers](https://mlsanity.com/paper/2202.07894/) (2022)
- [Adapting GPT, GPT-2 and BERT Language Models for Speech Recognition](https://mlsanity.com/paper/2108.07789/) (2021)
- [ASR Rescoring and Confidence Estimation with ELECTRA](https://mlsanity.com/paper/2110.01857/) (2021)

## Built on by (most cited first, 23 of 23)

- [HyPoradise: An Open Baseline for Generative Speech Recognition with Large Language Models](https://mlsanity.com/paper/2309.15701/) (2023)
- [Multi-stage Large Language Model Correction for Speech Recognition](https://mlsanity.com/paper/2310.11532/) (2023)
- [BERT Meets CTC: New Formulation of End-to-End Speech Recognition with Pre-trained Masked Language Model](https://mlsanity.com/paper/2210.16663/) (2022)
- [When Large Language Models Meet Speech: A Survey on Integration Approaches](https://mlsanity.com/paper/2502.19548/) (2025)
- [BECTRA: Transducer-based End-to-End ASR with BERT-Enhanced Encoder](https://mlsanity.com/paper/2211.00792/) (2022)
- [SALSA: Speedy ASR-LLM Synchronous Aggregation](https://mlsanity.com/paper/2408.16542/) (2024)
- [Exploring the Integration of Large Language Models into Automatic Speech Recognition Systems: An Empirical Study](https://mlsanity.com/paper/2307.06530/) (2023)
- [Contextual Biasing of Named-Entities with Large Language Models](https://mlsanity.com/paper/2309.00723/) (2023)
- [Understanding Shared Speech-Text Representations](https://mlsanity.com/paper/2304.14514/) (2023)
- [ProGRes: Prompted Generative Rescoring on ASR n-Best](https://mlsanity.com/paper/2409.00217/) (2024)
- [Delayed Fusion: Integrating Large Language Models into First-Pass Decoding in End-to-end Speech Recognition](https://mlsanity.com/paper/2501.09258/) (2025)
- [Towards interfacing large language models with ASR systems using confidence measures and prompting](https://mlsanity.com/paper/2407.21414/) (2024)
- [Robust ASR Error Correction with Conservative Data Filtering](https://mlsanity.com/paper/2407.13300/) (2024)
- [Drax: Speech Recognition with Discrete Flow Matching](https://mlsanity.com/paper/2510.04162/) (2025)
- [Masked Audio Text Encoders are Effective Multi-Modal Rescorers](https://mlsanity.com/paper/2305.07677/) (2023)
- [Multiple Representation Transfer from Large Language Models to End-to-End ASR Systems](https://mlsanity.com/paper/2309.04031/) (2023)
- [Harnessing the Zero-Shot Power of Instruction-Tuned Large Language Model in End-to-End Speech Recognition](https://mlsanity.com/paper/2309.10524/) (2023)
- [Correction Focused Language Model Training for Speech Recognition](https://mlsanity.com/paper/2310.11003/) (2023)
- [Speech Recognition Rescoring with Large Speech-Text Foundation Models](https://mlsanity.com/paper/2409.16654/) (2024)
- [On the N-gram Approximation of Pre-trained Language Models](https://mlsanity.com/paper/2306.06892/) (2023)
- [MTLM: Incorporating Bidirectional Text Information to Enhance Language Model Training in Speech Recognition Systems](https://mlsanity.com/paper/2502.10058/) (2025)
- [Rethinking Human-Aligned Evaluation: An Analysis of Semantic Metrics Beyond WER](https://mlsanity.com/paper/2609.21663/) (2026)
- [Jev Matches 7B Language Models for Speech-Neuroprosthesis Rescoring](https://mlsanity.com/paper/2609.33538/) (2026)

## Abstract (arXiv)

Large-scale language models (LLMs) such as GPT-2, BERT and RoBERTa have been successfully applied to ASR N-best rescoring. However, whether or how they can benefit competitive, near state-of-the-art ASR systems remains unexplored. In this study, we incorporate LLM rescoring into one of the most competitive ASR baselines: the Conformer-Transducer model. We demonstrate that consistent improvement is achieved by the LLM's bidirectionality, pretraining, in-domain finetuning and context augmentation. Furthermore, our lexical analysis sheds light on how each of these components may be contributing to the ASR performance.