mlsanityData to 7 Oct 2026

Listen Again and Choose the Right Answer: A New Paradigm for Automatic Speech Recognition with Large Language Models

arXiv 2405.10025 ↗First posted 16 May 2024cs.CL cs.AI cs.LGcited by 13

Not on the map. Why?

Abstract (arXiv)

Recent advances in large language models (LLMs) have promoted generative error correction (GER) for automatic speech recognition (ASR), which aims to predict the ground-truth transcription from the decoded N-best hypotheses. Thanks to the strong language generation ability of LLMs and rich information in the N-best list, GER shows great effectiveness in enhancing ASR results. However, it still suffers from two limitations: 1) LLMs are unaware of the source speech during GER, which may lead to results that are grammatically correct but violate the source speech content, 2) N-best hypotheses usually only vary in a few tokens, making it redundant to send all of them for GER, which could confuse LLM about which tokens to focus on and thus lead to increased miscorrection. In this paper, we propose ClozeGER, a new paradigm for ASR generative error correction. First, we introduce a multimodal LLM (i.e., SpeechGPT) to receive source speech as extra input to improve the fidelity of correction output. Then, we reformat GER as a cloze test with logits calibration to remove the input information redundancy and simplify GER with clear instructions. Experiments show that ClozeGER achieves a new breakthrough over vanilla GER on 9 popular ASR datasets.

Key facts. Cited by 13 papers in the map (data through 7 Oct 2026). Builds on 33 of its 64 references that are in the map. First posted on arXiv 16 May 2024.

Builds on

33 of 64 references.

References · 33

  1. 2017Attention Is All You Needcited by 59,932
  2. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  3. 2020Language Models are Few-Shot Learnerscited by 32,199
  4. 2022Training language models to follow instructions with human feedbackcited by 15,712
  5. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  6. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  7. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  8. 2020Measuring Massive Multitask Language Understandingcited by 7,768
  9. 2022Robust Speech Recognition via Large-Scale Weak Supervisioncited by 3,972
  10. 2023Video-LLaMA: An Instruction-tuned Audio-Visual Language Model for Video Understandingcited by 1,461
  11. 2019Common Voice: A Massively-Multilingual Speech Corpuscited by 1,358
  12. 2023Large Language Models are not Fair Evaluatorscited by 1,050
21 more
  1. 2023LLaMA-Adapter: Efficient Fine-tuning of Language Models with Zero-init Attentioncited by 776
  2. 2023AGIEval: A Human-Centric Benchmark for Evaluating Foundation Modelscited by 701
  3. 2023C-Eval: A Multi-Level Multi-Discipline Chinese Evaluation Suite for Foundation Modelscited by 694
  4. 2023LLaMA-Adapter V2: Parameter-Efficient Visual Instruction Modelcited by 602
  5. 2023SpeechGPT: Empowering Large Language Models with Intrinsic Cross-Modal Conversational Abilitiescited by 518
  6. 2023Large Language Models Are Not Robust Multiple Choice Selectorscited by 454
  7. 2023Large Language Models Sensitivity to The Order of Options in Multiple-Choice Questionscited by 249
  8. 2016Lip Reading Sentences in the Wildcited by 226
  9. 2018TED-LIUM 3: twice as much data and corpus repartition for experiments on speaker adaptationcited by 224
  10. 2023Prompting Large Language Models with Speech Recognition Abilitiescited by 139
  11. 2023X-LLM: Bootstrapping Advanced Large Language Models by Treating Multi-Modalities as Foreign Languagescited by 127
  12. 2023Generative Speech Recognition Error Correction with Large Language Models and Task-Activating Promptingcited by 71
  13. 2023N-best T5: Robust ASR Error Correction using Multiple Input Hypotheses and Constrained Decoding Spacecited by 41
  14. 2022Noise-robust Speech Recognition with 10 Minutes Unparalleled In-domain Datacited by 30
  15. 2021FastCorrect 2: Fast Error Correction on Multiple Candidates for Automatic Speech Recognitioncited by 27
  16. 2020Emotion recognition by fusing time synchronous and time asynchronous representationscited by 26
  17. 2024GenTranslate: Large Language Models are Generative Multilingual Speech and Machine Translatorscited by 22
  18. 2022Robust Data2vec: Noise-robust Speech Representation Learning for ASR by Combining Regression and Improved Contrastive Learningcited by 21
  19. 2023Investigating Uncertainty Calibration of Aligned Language Models under the Multiple-Choice Settingcited by 20
  20. 2023Wav2code: Restore Clean Speech Representations via Codebook Lookup for Noise-Robust ASRcited by 10
  21. 2023Unsupervised Noise adaptation using Data Simulationcited by 9

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 2; 2025: 6; 2026: 5.

13 papers.

Sort:

By area:

  1. 2024ASR Error Correction using Large Language Modelscited by 23
  2. 2025GEC-RAG: Improving Generative Error Correction via Retrieval-Augmented Generation for Automatic Speech Recognition Systemscited by 5
  3. 2025Better Pseudo-labeling with Multi-ASR Fusion and Error Correction by SpeechLLMcited by 5
  4. 2026Whisper-CD: Accurate Long-Form Speech Recognition using Multi-Negative Contrastive Decodingcited by 3
  5. 2025SepALM: Audio Language Models Are Error Correctors for Robust Speech Separationcited by 2
  6. 2025SUTA-LM: Bridging Test-Time Adaptation and Language Model Rescoring for Robust ASRcited by 1
  7. 2025Listening, Imagining & Refining: A Heuristic Optimized ASR Correction Framework with LLMscited by 1
  8. 2024Empowering Dysarthric Speech: Leveraging Advanced LLMs for Accurate Speech Correction and Multimodal Emotion Analysiscited by 0
  9. 2025Incorporating Error Level Noise Embedding for Improving LLM-Assisted Robustness in Persian Speech Recognitioncited by 0
  10. 2026Advancing LLM-based phoneme-to-grapheme for multilingual speech recognitioncited by 0
  11. 2026Symphony for Speech-to-Text: Supporting Real-Time Medical Voice Interfacescited by 0
  12. 2026Read What You Hear: Reference-Free Hypotheses Evaluation with Acoustic Discrepancycited by 0
1 more
  1. 2026agentic-ger: terminology recovery in long-form speech using global contextcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025LLM-based Generative Error Correction for Rare Words with Synthetic Data and Phonetic Context
  2. 2024Large Language Models are Efficient Learners of Noise-Robust Speech Recognition
  3. 2023Multi-stage Large Language Model Correction for Speech Recognition
  4. 2024Failing Forward: Improving Generative Error Correction for ASR with Synthetic Data and Retrieval Augmentation
  5. 2024Large Language Model Based Generative Error Correction: A Challenge and Baselines for Speech Recognition, Speaker Tagging, and Emotion Recognition
  6. 2026NIM4-ASR: Towards Efficient, Robust, and Customizable Real-Time LLM-Based ASR
  7. 2025Fewer Hallucinations, More Verification: A Three-Stage LLM-Based Framework for ASR Error Correction
  8. 2023Harnessing the Zero-Shot Power of Instruction-Tuned Large Language Model in End-to-End Speech Recognition