# From Hype to Insight: Rethinking Large Language Model Integration in Visual Speech Recognition

arXiv: https://arxiv.org/abs/2509.14880 · HTML: https://mlsanity.com/paper/2509.14880/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 18 Sep 2025.

## Built on by (most cited first, 4 of 4)

- [The Lipreading Gap: Do VSR Models Perceive Visual Speech Like Human Lipreaders?](https://mlsanity.com/paper/2606.07435/) (2026)
- [VALLR-Pin: Uncertainty-Factorized Visual Speech Recognition for Mandarin with Pinyin Guidance](https://mlsanity.com/paper/2512.20032/) (2025)
- [AVSRBench: A Multi-Condition AVSR Benchmark](https://mlsanity.com/paper/2609.10366/) (2026)
- [Candor-LR: A Dyadic Conversational Dataset for Audio-Visual Speech Recognition](https://mlsanity.com/paper/2609.10394/) (2026)

## Abstract (arXiv)

Advances in self-supervised encoders have improved Visual Speech Recognition (VSR). Recent approaches integrating these encoders with LLM decoders improves transcription accuracy; however, it remains unclear whether these gains stem from visual understanding or stronger language modeling. In this work, we systematically evaluate LLM decoders by freezing or selectively updating the visual encoder, scaling decoder size, comparing adaptation strategies and architectures, and varying training data across LRS2, LRS3, and their combination. Evaluation on LRS2, LRS3, and WildVSR shows that scaling and adaptation yield limited improvements, while combining datasets enhances generalization. Semantic analysis reveals that gains arise primarily from lexical rather than semantic processing. Our Llama-2-13B model trained on the combined set achieves 24.7% WER on LRS3 and 47.0% on WildVSR, establishing SOTA among models trained without additional supervision. Our findings indicate LLM decoders refine contextual reasoning rather than visual features, emphasizing the need for stronger visual encoders to drive meaningful progress.