mlsanityData to 7 Oct 2026

Two Heads Are Better Than One: Audio-Visual Speech Error Correction with Dual Hypotheses

arXiv 2510.13281 ↗First posted 15 Oct 2025eess.AS cs.CL cs.LGcited by 1

Not on the map. Why?

Abstract (arXiv)

This paper introduces a new paradigm for generative error correction (GER) framework in audio-visual speech recognition (AVSR) that reasons over modality-specific evidences directly in the language space. Our framework, DualHyp, empowers a large language model (LLM) to compose independent N-best hypotheses from separate automatic speech recognition (ASR) and visual speech recognition (VSR) models. To maximize the effectiveness of DualHyp, we further introduce RelPrompt, a noise-aware guidance mechanism that provides modality-grounded prompts to the LLM. RelPrompt offers the temporal reliability of each modality stream, guiding the model to dynamically switch its focus between ASR and VSR hypotheses for an accurate correction. Under various corruption scenarios, our framework attains up to 57.7% error rate gain on the LRS2 benchmark over standard ASR baseline, contrary to single-stream GER approaches that achieve only 10% gain. To facilitate research within our DualHyp framework, we release the code and the dataset comprising ASR and VSR hypotheses at https://github.com/sungnyun/dualhyp.

Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Builds on 35 of its 51 references that are in the map. First posted on arXiv 15 Oct 2025.

Builds on

35 of 51 references.

References · 35

  1. 2019Parameter-Efficient Transfer Learning for NLPcited by 4,195
  2. 2022Robust Speech Recognition via Large-Scale Weak Supervisioncited by 3,972
  3. 2024TinyLlama: An Open-Source Small Language Modelcited by 604
  4. 2015MUSAN: A Music, Speech, and Noise Corpuscited by 400
  5. 2018LRS3-TED: a large-scale dataset for visual speech recognitioncited by 301
  6. 2022Learning Audio-Visual Speech Representation by Masked Multimodal Cluster Predictioncited by 277
  7. 2020Transformer Transducer: A Streamable Speech Recognition Model with Transformer Encoders and RNN-T Losscited by 232
  8. 2016Lip Reading Sentences in the Wildcited by 226
  9. 2022Self-supervised Learning with Random-projection Quantizer for Speech Recognitioncited by 158
  10. 2023Prompting Large Language Models with Speech Recognition Abilitiescited by 139
  11. 2023Auto-AVSR: Audio-Visual Speech Recognition with Automatic Labelscited by 113
  12. 2023Connecting Speech Encoder and Large Language Model for ASRcited by 79
23 more
  1. 2024OWSM v3.1: Better and Faster Open Whisper-Style Speech Models based on E-Branchformercited by 74
  2. 2023Whispering LLaMA: A Cross-Modal Generative Error Correction Framework for Speech Recognitioncited by 56
  3. 2023How Do In-Context Examples Affect Compositional Generalization?cited by 51
  4. 2022Evaluating the Impact of Model Scale for Compositional Generalization in Semantic Parsingcited by 50
  5. 2023Prompting Large Language Models for Zero-Shot Domain Adaptation in Speech Recognitioncited by 48
  6. 2023Watch or Listen: Robust Audio-Visual Speech Recognition with Visual Corruption Modeling and Reliability Scoringcited by 42
  7. 2024Large Language Models are Strong Audio-Visual Speech Recognition Learnerscited by 40
  8. 2022Distinguishing Homophenes Using Multi-Head Visual-Audio Memory for Lip Readingcited by 39
  9. 2023MuAViC: A Multilingual Audio-Visual Corpus for Robust Speech Recognition and Robust Speech-to-Text Translationcited by 39
  10. 2024Where Visual Speech Meets Language: VSP-LLM Framework for Efficient and Context-Aware Visual Speech Processingcited by 29
  11. 2022Visual Context-driven Audio Feature Enhancement for Robust End-to-End Audio-Visual Speech Recognitioncited by 23
  12. 2021Learning Word-Level Confidence For Subword End-to-End ASRcited by 22
  13. 2024BRAVEn: Improving Self-Supervised Pre-training for Visual and Auditory Speech Recognitioncited by 20
  14. 2025MMS-LLaMA: Efficient LLM-based Audio-Visual Speech Recognition with Minimal Multimodal Speech Tokenscited by 19
  15. 2022Delving into High-Quality Synthetic Face Occlusion Segmentation Datasetscited by 11
  16. 2024XLAVS-R: Cross-Lingual Audio-Visual Speech Representation Learning for Noise-Robust Speech Perceptioncited by 11
  17. 2025Adaptive Audio-Visual Speech Recognition via Matryoshka-Based Multimodal LLMscited by 11
  18. 2025Scaling and Enhancing LLM-based AVSR: A Sparse Mixture of Projectors Approachcited by 10
  19. 2022RED-ACE: Robust Error Detection for ASR using Confidence Embeddingscited by 8
  20. 2024Efficient Training for Multilingual Visual Speech Recognition: Pre-training with Discretized Visual Speech Representationcited by 8
  21. 2025Multi-Task Corrupted Prediction for Learning Robust Audio-Visual Speech Representationcited by 5
  22. 2025Mixture of LoRA Experts with Multi-Modal and Multi-Granularity LLM Generative Error Correction for Accented Speech Recognitioncited by 4
  23. 2025Denoising GER: A Noise-Robust Generative Error Correction with LLM for Speech Recognitioncited by 3

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 1.

1 paper.

  1. 2026Voice Memory for Agentic Speech Recognitioncited by 0