mlsanityData to 7 Oct 2026

Whispering LLaMA: A Cross-Modal Generative Error Correction Framework for Speech Recognition

arXiv 2310.06434 ↗First posted 10 Oct 2023cs.CL cs.AI cs.MMcited by 56

Not on the map. Why?

Abstract (arXiv)

We introduce a new cross-modal fusion technique designed for generative error correction in automatic speech recognition (ASR). Our methodology leverages both acoustic information and external linguistic representations to generate accurate speech transcription contexts. This marks a step towards a fresh paradigm in generative error correction within the realm of n-best hypotheses. Unlike the existing ranking-based rescoring methods, our approach adeptly uses distinct initialization techniques and parameter-efficient algorithms to boost ASR performance derived from pre-trained speech and text models. Through evaluation across diverse ASR datasets, we evaluate the stability and reproducibility of our fusion technique, demonstrating its improved word error rate relative (WERR) performance in comparison to n-best hypotheses by relatively 37.66%. To encourage future research, we have made our code and pre-trained models open source at https://github.com/Srijith-rkr/Whispering-LLaMA.

Key facts. Cited by 56 papers in the map (data through 7 Oct 2026). Builds on 39 of its 50 references that are in the map. First posted on arXiv 10 Oct 2023.

Builds on

39 of 50 references.

References · 39

  1. 2017Attention Is All You Needcited by 59,932
  2. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  3. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  4. 2019Parameter-Efficient Transfer Learning for NLPcited by 4,195
  5. 2022Robust Speech Recognition via Large-Scale Weak Supervisioncited by 3,972
  6. 2017Sigmoid-Weighted Linear Units for Neural Network Function Approximation in Reinforcement Learningcited by 935
  7. 2023Multimodal Chain-of-Thought Reasoning in Language Modelscited by 740
  8. 2018ESPnet: End-to-End Speech Processing Toolkitcited by 554
  9. 2015Listen, Attend and Spellcited by 542
  10. 2015Deep Speech 2: End-to-End Speech Recognition in English and Mandarincited by 449
  11. 2021GigaSpeech: An Evolving, Multi-domain ASR Corpus with 10,000 Hours of Transcribed Audiocited by 391
  12. 2021Voice2Series: Reprogramming Acoustic Models for Time Series Classificationcited by 91
27 more
  1. 2023Generative Speech Recognition Error Correction with Large Language Models and Task-Activating Promptingcited by 71
  2. 2017Minimum Word Error Rate Training for Attention-based Sequence-to-Sequence Modelscited by 68
  3. 2019A spelling correction model for end-to-end speech recognitioncited by 63
  4. 2020Deliberation Model Based Two-Pass End-to-End Speech Recognitioncited by 57
  5. 2022The Dual Form of Neural Networks Revisited: Connecting Test Time Predictions to Training Patterns via Spotlights of Attentioncited by 45
  6. 2023Prompting the Hidden Talent of Web-Scale Speech Models for Zero-Shot Task Generalizationcited by 43
  7. 2023N-best T5: Robust ASR Error Correction using Multiple Input Hypotheses and Constrained Decoding Spacecited by 41
  8. 2023HyPoradise: An Open Baseline for Generative Speech Recognition with Large Language Modelscited by 41
  9. 2023Can Whisper perform speech-based in-context learning?cited by 35
  10. 2022RescoreBERT: Discriminative Speech Recognition Rescoring with BERTcited by 34
  11. 2023SpeechPrompt v2: Prompt Tuning for Speech Classification Taskscited by 31
  12. 2021Transformer Based Deliberation for Two-Pass Speech Recognitioncited by 29
  13. 2022Exploring Efficient-tuning Methods in Self-supervised Speech Modelscited by 26
  14. 2023From English to More Languages: Parameter-Efficient Model Reprogramming for Cross-Lingual Speech Recognitioncited by 23
  15. 2020Multi-task Language Modeling for Improving Speech Recognition of Rare Wordscited by 21
  16. 2020REDAT: Accent-Invariant Representation for End-to-End ASR by Domain Adversarial Training with Relabelingcited by 18
  17. 2022Low-Resource Music Genre Classification with Cross-Modal Neural Model Reprogrammingcited by 17
  18. 2021ASR Rescoring and Confidence Estimation with ELECTRAcited by 16
  19. 2019Audio-attention discriminative language model for ASR rescoringcited by 13
  20. 2021Domain-aware Neural Language Models for Speech Recognitioncited by 13
  21. 2022CHAPTER: Exploiting Convolutional Neural Network Adapters for Self-supervised Speech Modelscited by 13
  22. 2023Low-rank Adaptation of Large Language Model Rescoring for Parameter-Efficient Speech Recognitioncited by 13
  23. 2023How to Estimate Model Transferability of Pre-Trained Speech Models?cited by 10
  24. 2023JEIT: Joint End-to-End Model and Internal Language Model Training for Speech Recognitioncited by 9
  25. 2023A Parameter-Efficient Learning Approach to Arabic Dialect Identification with Pre-Trained General-Purpose Speech Modelcited by 8
  26. 2022Improving Fast-slow Encoder based Transducer with Streaming Deliberationcited by 5
  27. 2021Cross-Modal Transformer-Based Neural Correction Models for Automatic Speech Recognitioncited by 4

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 1; 2024: 27; 2025: 22; 2026: 6.

56 papers.

Sort:

By area:

  1. 2025OmniVinci: Enhancing Architecture and Data for Omni-Modal Understanding LLMcited by 53
  2. 2025DeSTA2.5-Audio: Toward General-Purpose Large Audio Language Model with Self-Generated Cross-Modal Alignmentcited by 47
  3. 2024Large Language Models are Efficient Learners of Noise-Robust Speech Recognitioncited by 30
  4. 2025Audio Large Language Models Can Be Descriptive Speech Quality Evaluatorscited by 25
  5. 2024Speech Translation with Speech Foundation Models and Large Language Models: What is There and What is Missing?cited by 23
  6. 2024It's Never Too Late: Fusing Acoustic Information into Large Language Models for Automatic Speech Recognitioncited by 22
  7. 2024GenTranslate: Large Language Models are Generative Multilingual Speech and Machine Translatorscited by 22
  8. 2025MMS-LLaMA: Efficient LLM-based Audio-Visual Speech Recognition with Minimal Multimodal Speech Tokenscited by 19
  9. 2024BESTOW: Efficient and Streamable Speech Language Model with the Best of Two Worlds in GPT and T5cited by 18
  10. 2024Speech Emotion Recognition with ASR Transcripts: A Comprehensive Study on Word Error Rate and Fusion Techniquescited by 15
  11. 2024Comparing Discrete and Continuous Space LLMs for Speech Recognitioncited by 15
  12. 2025TASTE: Text-Aligned Speech Tokenization and Embedding for Spoken Language Modelingcited by 15
38 more
  1. 2023Multi-stage Large Language Model Correction for Speech Recognitioncited by 13
  2. 2024Large Language Model Based Generative Error Correction: A Challenge and Baselines for Speech Recognition, Speaker Tagging, and Emotion Recognitioncited by 10
  3. 2024Crossmodal ASR Error Correction with Discrete Speech Unitscited by 9
  4. 2025When Large Language Models Meet Speech: A Survey on Integration Approachescited by 9
  5. 2024MEDSAGE: Enhancing Robustness of Medical Dialogue Summarization to ASR Errors with LLM-generated Synthetic Dialoguescited by 8
  6. 2024SALSA: Speedy ASR-LLM Synchronous Aggregationcited by 8
  7. 2024Simul-Whisper: Attention-Guided Streaming Whisper with Truncation Detectioncited by 7
  8. 2025From TOWER to SPIRE: Adding the Speech Modality to a Translation-Specialist LLMcited by 7
  9. 2024Using Large Language Model for End-to-End Chinese ASR and NERcited by 6
  10. 2024A Multimodal Approach to Device-Directed Speech Detection with Large Language Modelscited by 5
  11. 2024Revisiting ASR Error Correction with Specialized Modelscited by 5
  12. 2024FastAdaSP: Multitask-Adapted Efficient Inference for Large Speech Language Modelcited by 5
  13. 2024Let's Fuse Step by Step: A Generative Fusion Decoding Algorithm with LLMs for Robust and Instruction-Aware ASR and OCRcited by 4
  14. 2024Towards interfacing large language models with ASR systems using confidence measures and promptingcited by 4
  15. 2024Benchmarking Japanese Speech Recognition on ASR-LLM Setups with Multi-Pass Augmented Generative Error Correctioncited by 4
  16. 2024LA-RAG:Enhancing LLM-based ASR Accuracy with Retrieval-Augmented Generationcited by 4
  17. 2025Spoken Conversational Agents with Large Language Modelscited by 4
  18. 2024LLM-Driven Multimodal Opinion Expression Identificationcited by 3
  19. 2024Robust ASR Error Correction with Conservative Data Filteringcited by 3
  20. 2024Evolutionary Prompt Design for LLM-Based Post-ASR Error Correctioncited by 3
  21. 2024Large Language Model Should Understand Pinyin for Chinese ASR Error Correctioncited by 3
  22. 2025Linguistic Knowledge Transfer Learning for Speech Enhancementcited by 3
  23. 2026SpeechMapper: Speech-to-text Embedding Projector for LLMscited by 3
  24. 2025SepALM: Audio Language Models Are Error Correctors for Robust Speech Separationcited by 2
  25. 2025SpeechIQ: Speech-Agentic Intelligence Quotient Across Cognitive Levels in Voice Understanding by Large Language Modelscited by 2
  26. 2025Re-evaluating Minimum Bayes Risk Decoding for Automatic Speech Recognitioncited by 2
  27. 2024Llama-VITS: Enhancing TTS Synthesis with Semantic Awarenesscited by 1
  28. 2024Bridging Speech and Text: Enhancing ASR with Pinyin-to-Character Pre-training in LLMscited by 1
  29. 2024Enhancing AAC Software for Dysarthric Speakers in e-Health Settings: An Evaluation Using TORGOcited by 1
  30. 2025Listening and Seeing Again: Generative Error Correction for Audio-Visual Speech Recognitioncited by 1
  31. 2025PMF-CEC: Phoneme-augmented Multimodal Fusion for Context-aware ASR Error Correction with Error-specific Selective Decodingcited by 1
  32. 2025Confidence-Guided Error Correction for Disordered Speech Recognitioncited by 1
  33. 2025Two Heads Are Better Than One: Audio-Visual Speech Error Correction with Dual Hypothesescited by 1
  34. 2025Auto Review: Second Stage Error Detection for Highly Accurate Information Extraction from Phone Conversationscited by 0
  35. 2025Speech Recognition on TV Series with Video-guided Post-ASR Correctioncited by 0
  36. 2025VOX-KRIKRI: Unifying Speech and Language through Continuous Fusioncited by 0
  37. 2025Interactive In-Meeting Speaker Correction with Human Feedbackcited by 0
  38. 2025ELEGANCE: Efficient LLM Guidance for Audio-Visual Target Speech Extractioncited by 0

50 most cited of 56 All 56 (JSON)