mlsanityData to 7 Oct 2026

Multi-stage Large Language Model Correction for Speech Recognition

arXiv 2310.11532 ↗First posted 17 Oct 2023cs.CL eess.AScited by 13

Not on the map. Why?

Abstract (arXiv)

In this paper, we investigate the usage of large language models (LLMs) to improve the performance of competitive speech recognition systems. Different from previous LLM-based ASR error correction methods, we propose a novel multi-stage approach that utilizes uncertainty estimation of ASR outputs and reasoning capability of LLMs. Specifically, the proposed approach has two stages: the first stage is about ASR uncertainty estimation and exploits N-best list hypotheses to identify less reliable transcriptions; The second stage works on these identified transcriptions and performs LLM-based corrections. This correction task is formulated as a multi-step rule-based LLM reasoning process, which uses explicitly written rules in prompts to decompose the task into concrete reasoning steps. Our experimental results demonstrate the effectiveness of the proposed method by showing 10% ~ 20% relative improvement in WER over competitive ASR systems -- across multiple test domains and in zero-shot settings.

Key facts. Cited by 13 papers in the map (data through 7 Oct 2026). Builds on 37 of its 42 references that are in the map. First posted on arXiv 17 Oct 2023.

Builds on

37 of 42 references.

References · 37

  1. 2023GPT-4 Technical Reportcited by 18,132
  2. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  3. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  4. 2022Large Language Models are Zero-Shot Reasonerscited by 4,323
  5. 2022Robust Speech Recognition via Large-Scale Weak Supervisioncited by 3,972
  6. 2020wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representationscited by 3,853
  7. 2016A Baseline for Detecting Misclassified and Out-of-Distribution Examples in Neural Networkscited by 2,362
  8. 2022Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language modelscited by 1,970
  9. 2022Challenging BIG-Bench Tasks and Whether Chain-of-Thought Can Solve Themcited by 1,767
  10. 2022Rethinking the Role of Demonstrations: What Makes In-Context Learning Work?cited by 1,413
  11. 2022Least-to-Most Prompting Enables Complex Reasoning in Large Language Modelscited by 1,397
  12. 2019Common Voice: A Massively-Multilingual Speech Corpuscited by 1,358
25 more
  1. 2019SpecAugment: A Simple Data Augmentation Method for Automatic Speech Recognitioncited by 1,355
  2. 2022Decomposed Prompting: A Modular Approach for Solving Complex Taskscited by 554
  3. 2023Faith and Fate: Limits of Transformers on Compositionalitycited by 496
  4. 2020MLS: A Large-Scale Multilingual Dataset for Speech Researchcited by 467
  5. 2023Orca: Progressive Learning from Complex Explanation Traces of GPT-4cited by 369
  6. 2020Transformers as Soft Reasoners over Languagecited by 265
  7. 2018TED-LIUM 3: twice as much data and corpus repartition for experiments on speaker adaptationcited by 224
  8. 2020Pushing the Limits of Semi-Supervised Learning for Automatic Speech Recognitioncited by 185
  9. 2017An analysis of incorporating an external language model into a sequence-to-sequence modelcited by 109
  10. 2017Cold Fusion: Training Seq2Seq Models Together with Language Modelscited by 87
  11. 2021SpeechStew: Simply Mix All Available Speech Recognition Data to Train One Large Neural Networkcited by 83
  12. 2020Rethinking Evaluation in ASR: Are Our Models Robust Enough?cited by 72
  13. 2023Generative Speech Recognition Error Correction with Large Language Models and Task-Activating Promptingcited by 71
  14. 2021Understanding Softmax Confidence and Uncertaintycited by 62
  15. 2023Can Generative Large Language Models Perform ASR Error Correction?cited by 56
  16. 2023Whispering LLaMA: A Cross-Modal Generative Error Correction Framework for Speech Recognitioncited by 56
  17. 2023N-best T5: Robust ASR Error Correction using Multiple Input Hypotheses and Constrained Decoding Spacecited by 41
  18. 2023HyPoradise: An Open Baseline for Generative Speech Recognition with Large Language Modelscited by 41
  19. 2020Confidence Estimation for Attention-based Sequence-to-sequence Models for Speech Recognitioncited by 35
  20. 2019Improving sequence-to-sequence speech recognition training with on-the-fly data augmentationcited by 29
  21. 2022Effect and Analysis of Large-scale Language Model Rescoring on Competitive ASR Systemscited by 23
  22. 2021Learning Word-Level Confidence For Subword End-to-End ASRcited by 22
  23. 2021Adapting GPT, GPT-2 and BERT Language Models for Speech Recognitioncited by 16
  24. 2023Large-scale Language Model Rescoring on Long-form Datacited by 14
  25. 2022RED-ACE: Robust Error Detection for ASR using Confidence Embeddingscited by 8

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 4; 2025: 7; 2026: 2.

13 papers.

Sort:

By area:

  1. 2025Better Pseudo-labeling with Multi-ASR Fusion and Error Correction by SpeechLLMcited by 5
  2. 2024Blending LLMs into Cascaded Speech Translation: KIT's Offline Speech Translation System for IWSLT 2024cited by 4
  3. 2024Towards interfacing large language models with ASR systems using confidence measures and promptingcited by 4
  4. 2025LLM-based Generative Error Correction for Rare Words with Synthetic Data and Phonetic Contextcited by 3
  5. 2024PERL: Pinyin Enhanced Rephrasing Language Model for Chinese ASR N-best Error Correctioncited by 1
  6. 2025Optimizing Estonian TV Subtitles with Semi-supervised Learning and LLMscited by 1
  7. 2025Predicting Compact Phrasal Rewrites with Large Language Models for ASR Post Editingcited by 1
  8. 2025Confidence-Guided Error Correction for Disordered Speech Recognitioncited by 1
  9. 2024Empowering Dysarthric Speech: Leveraging Advanced LLMs for Accurate Speech Correction and Multimodal Emotion Analysiscited by 0
  10. 2025Understanding Zero-shot Rare Word Recognition Improvements Through LLM Integrationcited by 0
  11. 2025Improving Speech Recognition of Named Entities in Classroom Speech with LLM Revision and Phonetic-Semantic Contextcited by 0
  12. 2026RECOVER: Robust Entity Correction via agentic Orchestration of hypothesis Variants for Evidence-based Recoverycited by 0
1 more
  1. 2026Leveraging Fine-grained Error Correction in Korean Speech Recognition for Consultation Servicescited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Fewer Hallucinations, More Verification: A Three-Stage LLM-Based Framework for ASR Error Correction
  2. 2024Large Language Models are Efficient Learners of Noise-Robust Speech Recognition
  3. 2024Listen Again and Choose the Right Answer: A New Paradigm for Automatic Speech Recognition with Large Language Models
  4. 2026NIM4-ASR: Towards Efficient, Robust, and Customizable Real-Time LLM-Based ASR
  5. 2026Rethinking Entropy Allocation in LLM-based ASR: Understanding the Dynamics between Speech Encoders and LLMs
  6. 2026Enhancing Multilingual LLM-based ASR with Mixture of Experts and Dynamic Downsampling
  7. 2023Harnessing the Zero-Shot Power of Instruction-Tuned Large Language Model in End-to-End Speech Recognition
  8. 2025Customizing Speech Recognition Model with Large Language Model Feedback