mlsanityData to 14 Oct 2026

Generative Speech Recognition Error Correction with Large Language Models and Task-Activating Prompting

arXiv 2309.15649 ↗First posted 27 Sep 2023cs.CL cs.AI cs.LGcited by 72

Not on the map. Why?

Abstract (arXiv)

We explore the ability of large language models (LLMs) to act as speech recognition post-processors that perform rescoring and error correction. Our first focus is on instruction prompting to let LLMs perform these task without fine-tuning, for which we evaluate different prompting schemes, both zero- and few-shot in-context learning, and a novel task activation prompting method that combines causal instructions and demonstration to increase its context windows. Next, we show that rescoring only by in-context learning with frozen LLMs achieves results that are competitive with rescoring by domain-tuned LMs, using a pretrained first-pass recognition system and rescoring output on two out-of-domain tasks (ATIS and WSJ). By combining prompting techniques with fine-tuning we achieve error rates below the N-best oracle level, showcasing the generalization power of the LLMs.

Key facts. Cited by 72 papers in the map (data through 14 Oct 2026). Builds on 42 of its 55 references that are in the map. First posted on arXiv 27 Sep 2023.

Builds on

42 of 55 references.

References · 42

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  2. 2020Language Models are Few-Shot Learnerscited by 32,419
  3. 2022Training language models to follow instructions with human feedbackcited by 15,971
  4. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
  5. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,459
  6. 2022Large Language Models are Zero-Shot Reasonerscited by 4,374
  7. 2019Parameter-Efficient Transfer Learning for NLPcited by 4,236
  8. 2021Prefix-Tuning: Optimizing Continuous Prompts for Generationcited by 3,915
  9. 2017Deep reinforcement learning from human preferencescited by 3,912
  10. 2020wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representationscited by 3,905
  11. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,780
  12. 2016Pointer Sentinel Mixture Modelscited by 3,096
30 more
  1. 2019ALBERT: A Lite BERT for Self-supervised Learning of Language Representationscited by 2,790
  2. 2022BLOOM: A 176B-Parameter Open-Access Multilingual Language Modelcited by 1,984
  3. 2020Conformer: Convolution-augmented Transformer for Speech Recognitioncited by 1,863
  4. 2021Multitask Prompted Training Enables Zero-Shot Task Generalizationcited by 1,470
  5. 2022Rethinking the Role of Demonstrations: What Makes In-Context Learning Work?cited by 1,434
  6. 2021An Explanation of In-context Learning as Implicit Bayesian Inferencecited by 938
  7. 2022Fine-Tuning can Distort Pretrained Features and Underperform Out-of-Distributioncited by 739
  8. 2021Datasets: A Community Library for Natural Language Processingcited by 530
  9. 2019Libri-Light: A Benchmark for ASR with Limited or No Supervisioncited by 467
  10. 2021VoxPopuli: A Large-Scale Multilingual Speech Corpus for Representation Learning, Semi-Supervised Learning and Interpretationcited by 463
  11. 2021GigaSpeech: An Evolving, Multi-domain ASR Corpus with 10,000 Hours of Transcribed Audiocited by 399
  12. 2023Google USM: Scaling Automatic Speech Recognition Beyond 100 Languagescited by 231
  13. 2018A Simple Method for Commonsense Reasoningcited by 223
  14. 2021WARP: Word-level Adversarial ReProgrammingcited by 221
  15. 2022Why Can GPT Learn In-Context? Language Models Implicitly Perform Gradient Descent as Meta-Optimizerscited by 204
  16. 2017State-of-the-art Speech Recognition With Sequence-to-Sequence Modelscited by 196
  17. 2022Impact of Pretraining Term Frequencies on Few-Shot Reasoningcited by 142
  18. 2021Adapting Language Models for Zero-shot Learning by Meta-tuning on Dataset and Prompt Collectionscited by 128
  19. 2021Voice2Series: Reprogramming Acoustic Models for Time Series Classificationcited by 91
  20. 2017Minimum Word Error Rate Training for Attention-based Sequence-to-Sequence Modelscited by 68
  21. 2020Deep Shallow Fusion for RNN-T Personalizationcited by 64
  22. 2019A spelling correction model for end-to-end speech recognitioncited by 63
  23. 2021Scaling End-to-End Models for Large-Scale Multilingual ASRcited by 61
  24. 2023N-best T5: Robust ASR Error Correction using Multiple Input Hypotheses and Constrained Decoding Spacecited by 42
  25. 2022RescoreBERT: Discriminative Speech Recognition Rescoring with BERTcited by 34
  26. 2022WAVPROMPT: Towards Few-Shot Spoken Language Understanding with Frozen Language Modelscited by 26
  27. 2020Improving Readability for Automatic Speech Recognition Transcriptioncited by 22
  28. 2022SpeechPrompt: An Exploration of Prompt Tuning on Generative Spoken Language Model for Speech Processing Taskscited by 19
  29. 2022ASR Error Correction with Constrained Decoding on Operation Predictioncited by 6
  30. 2021Attention-based Contextual Language Model Adaptation for Speech Recognitioncited by 4

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 3; 2024: 27; 2025: 23; 2026: 19.

72 papers.

Sort:

By area:

  1. 2023SALM: Speech-augmented Language Model with In-context Learning for Speech Recognition and Translationcited by 76
  2. 2025OmniVinci: Enhancing Architecture and Data for Omni-Modal Understanding LLMcited by 59
  3. 2023Whispering LLaMA: A Cross-Modal Generative Error Correction Framework for Speech Recognitioncited by 56
  4. 2024Automatic Speech Recognition using Advanced Deep Learning Approaches: A surveycited by 50
  5. 2024Large Language Models are Efficient Learners of Noise-Robust Speech Recognitioncited by 31
  6. 2024It's Never Too Late: Fusing Acoustic Information into Large Language Models for Automatic Speech Recognitioncited by 22
  7. 2024GenTranslate: Large Language Models are Generative Multilingual Speech and Machine Translatorscited by 22
  8. 2025AudioJudge: Understanding What Works in Large Audio Model Based Speech Evaluationcited by 21
  9. 2024Chain-of-Thought Prompting for Speech Translationcited by 17
  10. 2024Performance evaluation of SLAM-ASR: The Good, the Bad, the Ugly, and the Way Forwardcited by 17
  11. 2024Listen Again and Choose the Right Answer: A New Paradigm for Automatic Speech Recognition with Large Language Modelscited by 14
  12. 2023Multi-stage Large Language Model Correction for Speech Recognitioncited by 13
38 more
  1. 2024Large Language Model Based Generative Error Correction: A Challenge and Baselines for Speech Recognition, Speaker Tagging, and Emotion Recognitioncited by 11
  2. 2024Failing Forward: Improving Generative Error Correction for ASR with Synthetic Data and Retrieval Augmentationcited by 10
  3. 2024Crossmodal ASR Error Correction with Discrete Speech Unitscited by 9
  4. 2025When Large Language Models Meet Speech: A Survey on Integration Approachescited by 9
  5. 2024Self-Taught Recognizer: Toward Unsupervised Adaptation for Speech Foundation Modelscited by 8
  6. 2024MEDSAGE: Enhancing Robustness of Medical Dialogue Summarization to ASR Errors with LLM-generated Synthetic Dialoguescited by 8
  7. 2024SALSA: Speedy ASR-LLM Synchronous Aggregationcited by 8
  8. 2026Text-only adaptation in LLM-based ASR through text denoisingcited by 8
  9. 2024ASR-EC Benchmark: Evaluating Large Language Models on Chinese ASR Error Correctioncited by 7
  10. 2025Fewer Hallucinations, More Verification: A Three-Stage LLM-Based Framework for ASR Error Correctioncited by 7
  11. 2025Better Pseudo-labeling with Multi-ASR Fusion and Error Correction by SpeechLLMcited by 6
  12. 2026The Interspeech 2026 Audio Reasoning Challenge: Evaluating Reasoning Process Quality for Audio Reasoning Models and Agentscited by 6
  13. 2024Revisiting ASR Error Correction with Specialized Modelscited by 5
  14. 2024Revise, Reason, and Recognize: LLM-Based Emotion Recognition via Emotion-Specific Prompts and ASR Error Correctioncited by 5
  15. 2025Delayed Fusion: Integrating Large Language Models into First-Pass Decoding in End-to-end Speech Recognitioncited by 5
  16. 2025Multi-Domain Audio Question Answering Benchmark Toward Acoustic Content Reasoningcited by 5
  17. 2026From Text to Voice: A Reproducible and Verifiable Framework for Evaluating Tool Calling LLM Agentscited by 5
  18. 2024Let's Fuse Step by Step: A Generative Fusion Decoding Algorithm with LLMs for Robust and Instruction-Aware ASR and OCRcited by 4
  19. 2024Evolutionary Prompt Design for LLM-Based Post-ASR Error Correctioncited by 4
  20. 2024Towards interfacing large language models with ASR systems using confidence measures and promptingcited by 4
  21. 2024Benchmarking Japanese Speech Recognition on ASR-LLM Setups with Multi-Pass Augmented Generative Error Correctioncited by 4
  22. 2024LA-RAG:Enhancing LLM-based ASR Accuracy with Retrieval-Augmented Generationcited by 4
  23. 2025Neural Spelling: A Spell-Based BCI System for Language Neural Decodingcited by 4
  24. 2025LLM-based Generative Error Correction for Rare Words with Synthetic Data and Phonetic Contextcited by 4
  25. 2025DeRAGEC: Denoising Named Entity Candidates with Synthetic Rationale for ASR Error Correctioncited by 4
  26. 2025Spoken Conversational Agents with Large Language Modelscited by 4
  27. 2024Robust ASR Error Correction with Conservative Data Filteringcited by 3
  28. 2024Large Language Model Should Understand Pinyin for Chinese ASR Error Correctioncited by 3
  29. 2024Semi-Supervised Cognitive State Classification from Speech with Multi-View Pseudo-Labelingcited by 3
  30. 2024Optimizing Contextual Speech Recognition Using Vector Quantization for Efficient Retrievalcited by 3
  31. 2025CantoASR: Prosody-Aware ASR-LALM Collaboration for Low-Resource Cantonesecited by 3
  32. 2026From Knowing to Doing Precisely: A General Self-Correction and Termination Framework for VLA modelscited by 3
  33. 2026Whisper-CD: Accurate Long-Form Speech Recognition using Multi-Negative Contrastive Decodingcited by 3
  34. 2024Recording for Eyes, Not Echoing to Ears: Contextualized Spoken-to-Written Conversion of ASR Transcriptscited by 2
  35. 2025SpeechIQ: Speech-Agentic Intelligence Quotient Across Cognitive Levels in Voice Understanding by Large Language Modelscited by 2
  36. 2025Re-evaluating Minimum Bayes Risk Decoding for Automatic Speech Recognitioncited by 2
  37. 2026Speech-Hands: A Self-Reflection Voice Agentic Approach to Speech Recognition and Audio Reasoning with Omni Perceptioncited by 2
  38. 2026Reducing Prompt Sensitivity in LLM-based Speech Recognition Through Learnable Projectioncited by 2

50 most cited of 72 All 72 (JSON)