# Diffusion Large Language Models for Visual Speech Recognition

arXiv: https://arxiv.org/abs/2605.28456 · HTML: https://mlsanity.com/paper/2605.28456/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 28 of its 40 references that are in the map.
- First posted on arXiv 27 May 2026.

## Builds on (most cited first, 28 of 28)

- [Deep Audio-Visual Speech Recognition](https://mlsanity.com/paper/1809.02108/) (2018)
- [LRS3-TED: a large-scale dataset for visual speech recognition](https://mlsanity.com/paper/1809.00496/) (2018)
- [Learning Audio-Visual Speech Representation by Masked Multimodal Cluster Prediction](https://mlsanity.com/paper/2201.02184/) (2022)
- [Scaling Diffusion Language Models via Adaptation from Autoregressive Models](https://mlsanity.com/paper/2410.17891/) (2024)
- [End-to-end Audio-visual Speech Recognition with Conformers](https://mlsanity.com/paper/2102.06657/) (2021)
- [LLaDA-V: Large Language Diffusion Models with Visual Instruction Tuning](https://mlsanity.com/paper/2505.16933/) (2025)
- [Auto-AVSR: Audio-Visual Speech Recognition with Automatic Labels](https://mlsanity.com/paper/2303.14307/) (2023)
- [Dimple: Discrete Diffusion Multimodal Large Language Model with Parallel Decoding](https://mlsanity.com/paper/2505.16990/) (2025)
- [LaViDa: A Large Diffusion Language Model for Multimodal Understanding](https://mlsanity.com/paper/2505.16839/) (2025)
- [Dream-Coder 7B: An Open Diffusion Language Model for Code](https://mlsanity.com/paper/2509.01142/) (2025)
- [LipNet: End-to-End Sentence-level Lipreading](https://mlsanity.com/paper/1611.01599/) (2016)
- [Recurrent Neural Network Transducer for Audio-Visual Speech Recognition](https://mlsanity.com/paper/1911.04890/) (2019)
- [Combining Residual Networks with LSTMs for Lipreading](https://mlsanity.com/paper/1703.04105/) (2017)
- [Sub-word Level Lip Reading With Visual Attention](https://mlsanity.com/paper/2110.07603/) (2021)
- [Jointly Learning Visual and Auditory Speech Representations from Raw Data](https://mlsanity.com/paper/2212.06246/) (2022)
- [Large Language Models are Strong Audio-Visual Speech Recognition Learners](https://mlsanity.com/paper/2409.12319/) (2024)
- [Hearing Lips: Improving Lip Reading by Distilling Speech Recognizers](https://mlsanity.com/paper/1911.11502/) (2019)
- [VATLM: Visual-Audio-Text Pre-Training with Unified Masked Prediction for Speech Representation Learning](https://mlsanity.com/paper/2211.11275/) (2022)
- [Where Visual Speech Meets Language: VSP-LLM Framework for Efficient and Context-Aware Visual Speech Processing](https://mlsanity.com/paper/2402.15151/) (2024)
- [Diffusion LLM with Native Variable Generation Lengths: Let [EOS] Lead the Way](https://mlsanity.com/paper/2510.24605/) (2025)
- [DreamOn: Diffusion Language Models For Code Infilling Beyond Fixed-size Canvas](https://mlsanity.com/paper/2602.01326/) (2026)
- [AKVSR: Audio Knowledge Empowered Visual Speech Recognition by Compressing Audio Knowledge of a Pretrained Model](https://mlsanity.com/paper/2308.07593/) (2023)
- [Lip Reading for Low-resource Languages by Learning and Combining General Speech Knowledge and Language-specific Knowledge](https://mlsanity.com/paper/2308.09311/) (2023)
- [Visual Speech Recognition for Languages with Limited Labeled Data using Automatic Labels from Whisper](https://mlsanity.com/paper/2309.08535/) (2023)
- [Do VSR Models Generalize Beyond LRS3?](https://mlsanity.com/paper/2311.14063/) (2023)
- [Decoding visemes: improving machine lipreading](https://mlsanity.com/paper/1710.01288/) (2017)
- [Improving Variable-Length Generation in Diffusion Language Models via Length Regularization](https://mlsanity.com/paper/2602.07546/) (2026)
- [Pay Attention to CTC: Fast and Robust Pseudo-Labelling for Unified Speech Recognition](https://mlsanity.com/paper/2602.19316/) (2026)

## Abstract (arXiv)

Existing Visual Speech Recognition (VSR) systems commonly rely on left-to-right autoregressive decoding, which can force premature decisions on visually ambiguous tokens before sufficient context is available. We propose DLLM-VSR, to the best of our knowledge, the first Diffusion Large Language Model (DLLM)-based VSR framework, formulating transcription as iterative masked denoising with flexible-order decoding. With confidence-based unmasking, DLLM-VSR commits high-confidence positions early and uses the committed tokens as bidirectional context to refine ambiguous ones. To adapt DLLMs to VSR, we introduce a two-stage masked-denoising training strategy that separates visual-to-text content alignment from length modeling. We further observe a performance gap compared with an upper-bound setting where the ground-truth transcript length is provided at inference, allowing the model to focus on transcript content decoding. To reduce this gap, we develop length-guided candidate decoding, which uses video duration to construct plausible transcript-length hypotheses and reranks the decoded candidates using length plausibility and decoding confidence. The proposed method achieves a 19.4% word error rate on LRS3, establishing state-of-the-art performance among methods using only LRS3 as labeled training data.