mlsanityData to 7 Oct 2026

Let's Fuse Step by Step: A Generative Fusion Decoding Algorithm with LLMs for Robust and Instruction-Aware ASR and OCR

arXiv 2405.14259 ↗First posted 23 May 2024cs.CL cs.AIcited by 4

Not on the map. Why?

Abstract (arXiv)

We propose "Generative Fusion Decoding" (GFD), a novel shallow fusion framework designed to integrate large language models (LLMs) into cross-modal text recognition systems for automatic speech recognition (ASR) and optical character recognition (OCR). We derive the necessary formulations to enable GFD to operate across mismatched token spaces of different models by calculating likelihood at the byte level, thereby enabling seamless fusion and synchronous progression during the decoding process. GFD is plug-and-play by design, making it readily compatible with various auto-regressive models without the need for any re-training. GFD proves effective for general ASR and OCR tasks through intermediate and frequent interactions with LLMs, surpassing cascaded methods in English and Mandarin benchmarks. In addition, GFD transfers in-context learning abilities of LLMs and allows for adaptive ASR in instruction-aware and long-context settings, yielding significant WER reductions of up to 17.7%.

Key facts. Cited by 4 papers in the map (data through 7 Oct 2026). Builds on 34 of its 47 references that are in the map. First posted on arXiv 23 May 2024.

Builds on

34 of 47 references.

References · 34

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2022Training language models to follow instructions with human feedbackcited by 15,712
  3. 2022Robust Speech Recognition via Large-Scale Weak Supervisioncited by 3,972
  4. 2020Learning to summarize from human feedbackcited by 2,562
  5. 2023PaLM-E: An Embodied Multimodal Language Modelcited by 2,037
  6. 2022PaLI: A Jointly-Scaled Multilingual Language-Image Modelcited by 762
  7. 2022FLEURS: Few-shot Learning Evaluation of Universal Representations of Speechcited by 458
  8. 2023Foundation Models for Decision Making: Problems, Methods, and Opportunitiescited by 160
  9. 2015On Using Monolingual Corpora in Neural Machine Translationcited by 149
  10. 2023Prompting Large Language Models with Speech Recognition Abilitiescited by 139
  11. 2017An analysis of incorporating an external language model into a sequence-to-sequence modelcited by 109
  12. 2022mSLAM: Massively multilingual joint pre-training for speech and textcited by 108
22 more
  1. 2021Contextualized Streaming End-to-End Speech Recognition with Trie-Based Deep Biasing and Shallow Fusioncited by 100
  2. 2023Provable Dynamic Fusion for Low-Quality Multimodal Datacited by 81
  3. 2023Whisper-AT: Noise-Robust Automatic Speech Recognizers are Also Strong General Audio Event Taggerscited by 72
  4. 2023Generative Speech Recognition Error Correction with Large Language Models and Task-Activating Promptingcited by 71
  5. 2021SLAM: A Unified Encoder for Speech and Language Modeling via Speech-Text Joint Pre-Trainingcited by 69
  6. 2019Two-Pass End-to-End Speech Recognitioncited by 62
  7. 2020Deliberation Model Based Two-Pass End-to-End Speech Recognitioncited by 57
  8. 2023Whispering LLaMA: A Cross-Modal Generative Error Correction Framework for Speech Recognitioncited by 56
  9. 2023On Uni-Modal Feature Learning in Supervised Multi-Modal Learningcited by 53
  10. 2023Prompting Large Language Models for Zero-Shot Domain Adaptation in Speech Recognitioncited by 48
  11. 2022RescoreBERT: Discriminative Speech Recognition Rescoring with BERTcited by 34
  12. 2019Deep Visual Template-Free Form Parsingcited by 24
  13. 2023Zero-shot Domain-sensitive Speech Recognition with Prompt-conditioning Fine-tuningcited by 13
  14. 2023Low-rank Adaptation of Large Language Model Rescoring for Parameter-Efficient Speech Recognitioncited by 13
  15. 2023Investigating Zero-Shot Generalizability on Mandarin-English Code-Switched ASR and Speech-to-text Translation of Recent Foundation Models with Self-Supervision and Weak Supervisioncited by 13
  16. 2024Multilingual and Fully Non-Autoregressive ASR with Large Language Model Fusion: A Comprehensive Studycited by 12
  17. 2019State-of-the-Art Speech Recognition Using Multi-Stream Self-Attention With Dilated 1D Convolutionscited by 11
  18. 2022g2pW: A Conditional Weighted Softmax BERT for Polyphone Disambiguation in Mandarincited by 7
  19. 2024Using Large Language Model for End-to-End Chinese ASR and NERcited by 6
  20. 2022A Likelihood Ratio based Domain Adaptation Method for E2E Modelscited by 5
  21. 2021Detecting English Speech in the Air Traffic Control Voice Communicationcited by 4
  22. 2023Improving Rare Words Recognition through Homophone Extension and Unified Writing for Low-resource Cantonese Speech Recognitioncited by 1

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 4.

4 papers.

  1. 2026Measuring the Redundancy of Decoder Layers in SpeechLLMscited by 0
  2. 2026Voice Memory for Agentic Speech Recognitioncited by 0
  3. 2026Anchoring Speech with Semantics: A Multimodal Adapter Mechanism for Automatic Speech Recognition in Low-Resource Languagescited by 0
  4. 2026Listen to the Latents: Self-Correcting Speech Recognition in Large Audio Language Models Through Hidden-State Interactionscited by 0