# N-best T5: Robust ASR Error Correction using Multiple Input Hypotheses and Constrained Decoding Space

arXiv: https://arxiv.org/abs/2303.00456 · HTML: https://mlsanity.com/paper/2303.00456/

## Key facts

- Cited by 42 papers in the map (data through 14 Oct 2026).
- Builds on 16 of its 29 references that are in the map.
- First posted on arXiv 1 Mar 2023.

## Builds on (most cited first, 16 of 16)

- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [Robust Speech Recognition via Large-Scale Weak Supervision](https://mlsanity.com/paper/2212.04356/) (2022)
- [Conformer: Convolution-augmented Transformer for Speech Recognition](https://mlsanity.com/paper/2005.08100/) (2020)
- [SpecAugment: A Simple Data Augmentation Method for Automatic Speech Recognition](https://mlsanity.com/paper/1904.08779/) (2019)
- [Plug and Play Language Models: A Simple Approach to Controlled Text Generation](https://mlsanity.com/paper/1912.02164/) (2019)
- [ESPnet: End-to-End Speech Processing Toolkit](https://mlsanity.com/paper/1804.00015/) (2018)
- [Listen, Attend and Spell](https://mlsanity.com/paper/1508.01211/) (2015)
- [A spelling correction model for end-to-end speech recognition](https://mlsanity.com/paper/1902.07178/) (2019)
- [Correction of Automatic Speech Recognition with Transformer Sequence-to-sequence Model](https://mlsanity.com/paper/1910.10697/) (2019)
- [Joint Contextual Modeling for ASR Correction and Language Understanding](https://mlsanity.com/paper/2002.00750/) (2020)
- [FastCorrect 2: Fast Error Correction on Multiple Candidates for Automatic Speech Recognition](https://mlsanity.com/paper/2109.14420/) (2021)
- [Less Is More: Improved RNN-T Decoding Using Limited Label Context and Path Merging](https://mlsanity.com/paper/2012.06749/) (2020)
- [BART based semantic correction for Mandarin automatic speech recognition system](https://mlsanity.com/paper/2104.05507/) (2021)
- [Error Correction in ASR using Sequence-to-Sequence Models](https://mlsanity.com/paper/2202.01157/) (2022)
- [Lattice Transformer for Speech Translation](https://mlsanity.com/paper/1906.05551/) (2019)
- [Automatic Spelling Correction with Transformer for CTC-based End-to-End Speech Recognition](https://mlsanity.com/paper/1904.10045/) (2019)

## Built on by (most cited first, 30 of 42)

- [SALM: Speech-augmented Language Model with In-context Learning for Speech Recognition and Translation](https://mlsanity.com/paper/2310.09424/) (2023)
- [Generative Speech Recognition Error Correction with Large Language Models and Task-Activating Prompting](https://mlsanity.com/paper/2309.15649/) (2023)
- [Can Generative Large Language Models Perform ASR Error Correction?](https://mlsanity.com/paper/2307.04172/) (2023)
- [Whispering LLaMA: A Cross-Modal Generative Error Correction Framework for Speech Recognition](https://mlsanity.com/paper/2310.06434/) (2023)
- [HyPoradise: An Open Baseline for Generative Speech Recognition with Large Language Models](https://mlsanity.com/paper/2309.15701/) (2023)
- [Large Language Models are Efficient Learners of Noise-Robust Speech Recognition](https://mlsanity.com/paper/2401.10446/) (2024)
- [ASR Error Correction using Large Language Models](https://mlsanity.com/paper/2409.09554/) (2024)
- [GenTranslate: Large Language Models are Generative Multilingual Speech and Machine Translators](https://mlsanity.com/paper/2402.06894/) (2024)
- [Listen Again and Choose the Right Answer: A New Paradigm for Automatic Speech Recognition with Large Language Models](https://mlsanity.com/paper/2405.10025/) (2024)
- [Multi-stage Large Language Model Correction for Speech Recognition](https://mlsanity.com/paper/2310.11532/) (2023)
- [Large Language Model Based Generative Error Correction: A Challenge and Baselines for Speech Recognition, Speaker Tagging, and Emotion Recognition](https://mlsanity.com/paper/2409.09785/) (2024)
- [Failing Forward: Improving Generative Error Correction for ASR with Synthetic Data and Retrieval Augmentation](https://mlsanity.com/paper/2410.13198/) (2024)
- [ASR-EC Benchmark: Evaluating Large Language Models on Chinese ASR Error Correction](https://mlsanity.com/paper/2412.03075/) (2024)
- [ProGRes: Prompted Generative Rescoring on ASR n-Best](https://mlsanity.com/paper/2409.00217/) (2024)
- [Revisiting ASR Error Correction with Specialized Models](https://mlsanity.com/paper/2405.15216/) (2024)
- [Delayed Fusion: Integrating Large Language Models into First-Pass Decoding in End-to-end Speech Recognition](https://mlsanity.com/paper/2501.09258/) (2025)
- [GEC-RAG: Improving Generative Error Correction via Retrieval-Augmented Generation for Automatic Speech Recognition Systems](https://mlsanity.com/paper/2501.10734/) (2025)
- [Evolutionary Prompt Design for LLM-Based Post-ASR Error Correction](https://mlsanity.com/paper/2407.16370/) (2024)
- [LegoSLM: Connecting LLM with Speech Encoder using CTC Posteriors](https://mlsanity.com/paper/2505.11352/) (2025)
- [Adapting an Unadaptable ASR System](https://mlsanity.com/paper/2306.01208/) (2023)
- [Robust ASR Error Correction with Conservative Data Filtering](https://mlsanity.com/paper/2407.13300/) (2024)
- [Full-text Error Correction for Chinese Speech Recognition with Large Language Model](https://mlsanity.com/paper/2409.07790/) (2024)
- [MathSpeech: Leveraging Small LMs for Accurate Conversion in Mathematical Speech-to-Formula](https://mlsanity.com/paper/2412.15655/) (2024)
- [CantoASR: Prosody-Aware ASR-LALM Collaboration for Low-Resource Cantonese](https://mlsanity.com/paper/2511.04139/) (2025)
- [Harnessing the Zero-Shot Power of Instruction-Tuned Large Language Model in End-to-End Speech Recognition](https://mlsanity.com/paper/2309.10524/) (2023)
- [Error Correction by Paying Attention to Both Acoustic and Confidence References for Automatic Speech Recognition](https://mlsanity.com/paper/2407.12817/) (2024)
- [PERL: Pinyin Enhanced Rephrasing Language Model for Chinese ASR N-best Error Correction](https://mlsanity.com/paper/2412.03230/) (2024)
- [FlanEC: Exploring Flan-T5 for Post-ASR Error Correction](https://mlsanity.com/paper/2501.12979/) (2025)
- [Chain of Correction for Full-text Speech Recognition with Large Language Models](https://mlsanity.com/paper/2504.01519/) (2025)
- [Speech-to-LaTeX: New Models and Datasets for Converting Spoken Equations and Sentences](https://mlsanity.com/paper/2508.03542/) (2025)

## Abstract (arXiv)

Error correction models form an important part of Automatic Speech Recognition (ASR) post-processing to improve the readability and quality of transcriptions. Most prior works use the 1-best ASR hypothesis as input and therefore can only perform correction by leveraging the context within one sentence. In this work, we propose a novel N-best T5 model for this task, which is fine-tuned from a T5 model and utilizes ASR N-best lists as model input. By transferring knowledge from the pre-trained language model and obtaining richer information from the ASR decoding space, the proposed approach outperforms a strong Conformer-Transducer baseline. Another issue with standard error correction is that the generation process is not well-guided. To address this a constrained decoding process, either based on the N-best list or an ASR lattice, is used which allows additional information to be propagated.