# Towards Estimating the Upper Bound of Visual-Speech Recognition: The Visual Lip-Reading Feasibility Database

arXiv: https://arxiv.org/abs/1704.08028 · HTML: https://mlsanity.com/paper/1704.08028/

## Key facts

- Cited by 13 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 26 Apr 2017.

## Built on by (most cited first, 13 of 13)

- [Deep Learning for Visual Speech Analysis: A Survey](https://mlsanity.com/paper/2205.10839/) (2022)
- [A cappella: Audio-visual Singing Voice Separation](https://mlsanity.com/paper/2104.09946/) (2021)
- [LIP-RTVE: An Audiovisual Database for Continuous Spanish in the Wild](https://mlsanity.com/paper/2311.12457/) (2023)
- [Advances and Challenges in Deep Lip Reading](https://mlsanity.com/paper/2110.07879/) (2021)
- [A Multi-Purpose Audio-Visual Corpus for Multi-Modal Persian Speech Recognition: the Arman-AV Dataset](https://mlsanity.com/paper/2301.10180/) (2023)
- [Bridge to Non-Barrier Communication: Gloss-Prompted Fine-grained Cued Speech Gesture Generation with Diffusion Model](https://mlsanity.com/paper/2404.19277/) (2024)
- [The Lipreading Gap: Do VSR Models Perceive Visual Speech Like Human Lipreaders?](https://mlsanity.com/paper/2606.07435/) (2026)
- [Tailored Design of Audio-Visual Speech Recognition Models using Branchformers](https://mlsanity.com/paper/2407.06606/) (2024)
- [Analysis of Visual Features for Continuous Lipreading in Spanish](https://mlsanity.com/paper/2311.12468/) (2023)
- [AnnoTheia: A Semi-Automatic Annotation Toolkit for Audio-Visual Speech Technologies](https://mlsanity.com/paper/2402.13152/) (2024)
- [ViCocktail: Automated Multi-Modal Data Collection for Vietnamese Audio-Visual Speech Recognition](https://mlsanity.com/paper/2506.04635/) (2025)
- [Speaker-Adapted End-to-End Visual Speech Recognition for Continuous Spanish](https://mlsanity.com/paper/2311.12480/) (2023)
- [Evaluation of End-to-End Continuous Spanish Lipreading in Different Data Conditions](https://mlsanity.com/paper/2502.00464/) (2025)

## Abstract (arXiv)

Speech is the most used communication method between humans and it involves the perception of auditory and visual channels. Automatic speech recognition focuses on interpreting the audio signals, although the video can provide information that is complementary to the audio. Exploiting the visual information, however, has proven challenging. On one hand, researchers have reported that the mapping between phonemes and visemes (visual units) is one-to-many because there are phonemes which are visually similar and indistinguishable between them. On the other hand, it is known that some people are very good lip-readers (e.g: deaf people). We study the limit of visual only speech recognition in controlled conditions. With this goal, we designed a new database in which the speakers are aware of being read and aim to facilitate lip-reading. In the literature, there are discrepancies on whether hearing-impaired people are better lip-readers than normal-hearing people. Then, we analyze if there are differences between the lip-reading abilities of 9 hearing-impaired and 15 normal-hearing people. Finally, human abilities are compared with the performance of a visual automatic speech recognition system. In our tests, hearing-impaired participants outperformed the normal-hearing participants but without reaching statistical significance. Human observers were able to decode 44% of the spoken message. In contrast, the visual only automatic system achieved 20% of word recognition rate. However, if we repeat the comparison in terms of phonemes both obtained very similar recognition rates, just above 50%. This suggests that the gap between human lip-reading and automatic speech-reading might be more related to the use of context than to the ability to interpret mouth appearance.