# Visual-Only Recognition of Normal, Whispered and Silent Speech

arXiv: https://arxiv.org/abs/1802.06399 · HTML: https://mlsanity.com/paper/1802.06399/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 18 Feb 2018.

## Built on by (most cited first, 7 of 7)

- [Visual Speech Recognition for Multiple Languages in the Wild](https://mlsanity.com/paper/2202.13084/) (2022)
- [A Survey on Deep Multi-modal Learning for Body Language Recognition and Generation](https://mlsanity.com/paper/2308.08849/) (2023)
- [Audio-video fusion strategies for active speaker detection in meetings](https://mlsanity.com/paper/2206.10411/) (2022)
- [Improving the Gap in Visual Speech Recognition Between Normal and Silent Speech Based on Metric Learning](https://mlsanity.com/paper/2305.14203/) (2023)
- [Automatic Speech Recognition And Limited Vocabulary: A Survey](https://mlsanity.com/paper/2108.10254/) (2021)
- [Cross-Modal Masking for Robust Silent Speech Synthesis Using sEMG and Lipreading](https://mlsanity.com/paper/2606.09667/) (2026)
- [Personalized Korean Lipreading as Visual Speech Recognition: Transfer, Census and Adaptation on OLKAVS](https://mlsanity.com/paper/2609.28988/) (2026)

## Abstract (arXiv)

Silent speech interfaces have been recently proposed as a way to enable communication when the acoustic signal is not available. This introduces the need to build visual speech recognition systems for silent and whispered speech. However, almost all the recently proposed systems have been trained on vocalised data only. This is in contrast with evidence in the literature which suggests that lip movements change depending on the speech mode. In this work, we introduce a new audiovisual database which is publicly available and contains normal, whispered and silent speech. To the best of our knowledge, this is the first study which investigates the differences between the three speech modes using the visual modality only. We show that an absolute decrease in classification rate of up to 3.7% is observed when training and testing on normal and whispered, respectively, and vice versa. An even higher decrease of up to 8.5% is reported when the models are tested on silent speech. This reveals that there are indeed visual differences between the 3 speech modes and the common assumption that vocalized training data can be used directly to train a silent speech recognition system may not be true.