mlsanityData to 14 Oct 2026

Do VSR Models Generalize Beyond LRS3?

arXiv 2311.14063 ↗First posted 23 Nov 2023cs.CV cs.CL cs.LGcited by 9

Not on the map. Why?

Abstract (arXiv)

The Lip Reading Sentences-3 (LRS3) benchmark has primarily been the focus of intense research in visual speech recognition (VSR) during the last few years. As a result, there is an increased risk of overfitting to its excessively used test set, which is only one hour duration. To alleviate this issue, we build a new VSR test set named WildVSR, by closely following the LRS3 dataset creation processes. We then evaluate and analyse the extent to which the current VSR models generalize to the new test data. We evaluate a broad range of publicly available VSR models and find significant drops in performance on our test set, compared to their corresponding LRS3 results. Our results suggest that the increase in word error rates is caused by the models inability to generalize to slightly harder and in the wild lip sequences than those found in the LRS3 test set. Our new test benchmark is made public in order to enable future research towards more robust VSR models.

Key facts. Cited by 9 papers in the map (data through 14 Oct 2026). Builds on 24 of its 47 references that are in the map. First posted on arXiv 23 Nov 2023.

Builds on

24 of 47 references.

References · 24

  1. 2020Scaling Laws for Neural Language Modelscited by 6,586
  2. 2022Robust Speech Recognition via Large-Scale Weak Supervisioncited by 4,057
  3. 2020wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representationscited by 3,905
  4. 2018VoxCeleb2: Deep Speaker Recognitioncited by 875
  5. 2020A Survey on Contrastive Self-supervised Learningcited by 554
  6. 2018LRS3-TED: a large-scale dataset for visual speech recognitioncited by 308
  7. 2022Learning Audio-Visual Speech Representation by Masked Multimodal Cluster Predictioncited by 287
  8. 2016Lip Reading Sentences in the Wildcited by 228
  9. 2021End-to-end Audio-visual Speech Recognition with Conformerscited by 151
  10. 2023Auto-AVSR: Audio-Visual Speech Recognition with Automatic Labelscited by 114
  11. 2022Visual Speech Recognition for Multiple Languages in the Wildcited by 95
  12. 2019ASR is all you need: cross-modal distillation for lip readingcited by 88
12 more
  1. 2022Robust Self-Supervised Audio-Visual Speech Recognitioncited by 75
  2. 2019Recurrent Neural Network Transducer for Audio-Visual Speech Recognitioncited by 74
  3. 2021Sub-word Level Lip Reading With Visual Attentioncited by 63
  4. 2022Visual Speech-Aware Perceptual 3D Facial Expression Reconstruction from Videoscited by 50
  5. 2022Jointly Learning Visual and Auditory Speech Representations from Raw Datacited by 47
  6. 2021YOLO5Face: Why Reinventing a Face Detectorcited by 44
  7. 2021LiRA: Learning Visual Speech Representations from Audio through Self-supervisioncited by 44
  8. 2022VATLM: Visual-Audio-Text Pre-Training with Unified Masked Prediction for Speech Representation Learningcited by 37
  9. 2022Training Strategies for Improved Lip-readingcited by 25
  10. 2023SynthVSR: Scaling Up Visual Speech Recognition With Synthetic Supervisioncited by 21
  11. 2023Conformers are All You Need for Visual Speech Recognitioncited by 9
  12. 2021Contrastive Learning of Global-Local Video Representationscited by 6

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 3; 2025: 0; 2026: 6.

9 papers.

  1. 2024Unified Speech Recognition: A Single Model for Auditory, Visual, and Audiovisual Inputscited by 15
  2. 2026The Lipreading Gap: Do VSR Models Perceive Visual Speech Like Human Lipreaders?cited by 3
  3. 2024ViSpeR: Multilingual Audio-Visual Speech Recognitioncited by 2
  4. 2024MSRS: Training Multimodal Speech Recognition Models from Scratch with Sparse Mask Optimizationcited by 2
  5. 2026Pay Attention to CTC: Fast and Robust Pseudo-Labelling for Unified Speech Recognitioncited by 1
  6. 2026Diffusion Large Language Models for Visual Speech Recognitioncited by 0
  7. 2026AVSRBench: A Multi-Condition AVSR Benchmarkcited by 0
  8. 2026Candor-LR: A Dyadic Conversational Dataset for Audio-Visual Speech Recognitioncited by 0
  9. 2026Learning Visual Speech Representations via Cross-Modal Distillation and Joint Face-Lip Modelingcited by 0