# ProGRes: Prompted Generative Rescoring on ASR n-Best

arXiv: https://arxiv.org/abs/2409.00217 · HTML: https://mlsanity.com/paper/2409.00217/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- Builds on 25 of its 41 references that are in the map.
- First posted on arXiv 30 Aug 2024.

## Builds on (most cited first, 25 of 25)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [LLaMA: Open and Efficient Foundation Language Models](https://mlsanity.com/paper/2302.13971/) (2023)
- [wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representations](https://mlsanity.com/paper/2006.11477/) (2020)
- [Neural Machine Translation of Rare Words with Subword Units](https://mlsanity.com/paper/1508.07909/) (2015)
- [Conformer: Convolution-augmented Transformer for Speech Recognition](https://mlsanity.com/paper/2005.08100/) (2020)
- [Common Voice: A Massively-Multilingual Speech Corpus](https://mlsanity.com/paper/1912.06670/) (2019)
- [ESPnet: End-to-End Speech Processing Toolkit](https://mlsanity.com/paper/1804.00015/) (2018)
- [SpeechBrain: A General-Purpose Speech Toolkit](https://mlsanity.com/paper/2106.04624/) (2021)
- [ESPnet-TTS: Unified, Reproducible, and Integratable Open Source End-to-End Text-to-Speech Toolkit](https://mlsanity.com/paper/1910.10909/) (2019)
- [minicons: Enabling Flexible Behavioral and Representational Analyses of Transformer Language Models](https://mlsanity.com/paper/2203.13112/) (2022)
- [Can Generative Large Language Models Perform ASR Error Correction?](https://mlsanity.com/paper/2307.04172/) (2023)
- [N-best T5: Robust ASR Error Correction using Multiple Input Hypotheses and Constrained Decoding Space](https://mlsanity.com/paper/2303.00456/) (2023)
- [RescoreBERT: Discriminative Speech Recognition Rescoring with BERT](https://mlsanity.com/paper/2202.01094/) (2022)
- [TorchAudio 2.1: Advancing speech recognition, self-supervised learning, and audio processing components for PyTorch](https://mlsanity.com/paper/2310.17864/) (2023)
- [Effect and Analysis of Large-scale Language Model Rescoring on Competitive ASR Systems](https://mlsanity.com/paper/2204.00212/) (2022)
- [The CHiME-7 DASR Challenge: Distant Meeting Transcription with Multiple Devices in Diverse Scenarios](https://mlsanity.com/paper/2306.13734/) (2023)
- [End-to-end model for named entity recognition from speech without paired training data](https://mlsanity.com/paper/2204.00803/) (2022)
- [Recent Progresses in Deep Learning based Acoustic Models (Updated)](https://mlsanity.com/paper/1804.09298/) (2018)
- [Correcting Automated and Manual Speech Transcription Errors using Warped Language Models](https://mlsanity.com/paper/2103.14580/) (2021)
- [Are LLMs Robust for Spoken Dialogues?](https://mlsanity.com/paper/2401.02297/) (2024)
- [Contaminated speech training methods for robust DNN-HMM distant speech recognition](https://mlsanity.com/paper/1710.03538/) (2017)
- [Transformer-based Model for ASR N-Best Rescoring and Rewriting](https://mlsanity.com/paper/2406.08207/) (2024)
- [Deep Learning for Distant Speech Recognition](https://mlsanity.com/paper/1712.06086/) (2017)

## Built on by (most cited first, 6 of 6)

- [Improving Child Speech Recognition and Reading Mistake Detection by Using Prompts](https://mlsanity.com/paper/2506.11079/) (2025)
- [Re-evaluating Minimum Bayes Risk Decoding for Automatic Speech Recognition](https://mlsanity.com/paper/2510.19471/) (2025)
- [VALLR-Pin: Uncertainty-Factorized Visual Speech Recognition for Mandarin with Pinyin Guidance](https://mlsanity.com/paper/2512.20032/) (2025)
- [Read What You Hear: Reference-Free Hypotheses Evaluation with Acoustic Discrepancy](https://mlsanity.com/paper/2606.04680/) (2026)
- [Contrastive Training with LLM-generated Near-Misses for Robust Code-Switching Speech Recognition](https://mlsanity.com/paper/2606.06985/) (2026)
- [Select or Rewrite: Oracle-Guided Fine-Tuning for Generative ASR Post-Editing from N-Best Hypotheses](https://mlsanity.com/paper/2610.15056/) (2026)

## Abstract (arXiv)

Large Language Models (LLMs) have shown their ability to improve the performance of speech recognizers by effectively rescoring the n-best hypotheses generated during the beam search process. However, the best way to exploit recent generative instruction-tuned LLMs for hypothesis rescoring is still unclear. This paper proposes a novel method that uses instruction-tuned LLMs to dynamically expand the n-best speech recognition hypotheses with new hypotheses generated through appropriately-prompted LLMs. Specifically, we introduce a new zero-shot method for ASR n-best rescoring, which combines confidence scores, LLM sequence scoring, and prompt-based hypothesis generation. We compare Llama-3-Instruct, GPT-3.5 Turbo, and GPT-4 Turbo as prompt-based generators with Llama-3 as sequence scorer LLM. We evaluated our approach using different speech recognizers and observed significant relative improvement in the word error rate (WER) ranging from 5% to 25%.