# Improving the Gap in Visual Speech Recognition Between Normal and Silent Speech Based on Metric Learning

arXiv: https://arxiv.org/abs/2305.14203 · HTML: https://mlsanity.com/paper/2305.14203/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 11 of its 27 references that are in the map.
- First posted on arXiv 23 May 2023.

## Builds on (most cited first, 11 of 11)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Deep Audio-Visual Speech Recognition](https://mlsanity.com/paper/1809.02108/) (2018)
- [LRS3-TED: a large-scale dataset for visual speech recognition](https://mlsanity.com/paper/1809.00496/) (2018)
- [Lip Reading Sentences in the Wild](https://mlsanity.com/paper/1611.05358/) (2016)
- [Visual Speech Recognition for Multiple Languages in the Wild](https://mlsanity.com/paper/2202.13084/) (2022)
- [Sub-word Level Lip Reading With Visual Attention](https://mlsanity.com/paper/2110.07603/) (2021)
- [Large-Scale Visual Speech Recognition](https://mlsanity.com/paper/1807.05162/) (2018)
- [Improving Speaker-Independent Lipreading with Domain-Adversarial Training](https://mlsanity.com/paper/1708.01565/) (2017)
- [Visual-Only Recognition of Normal, Whispered and Silent Speech](https://mlsanity.com/paper/1802.06399/) (2018)
- [Lip reading using external viseme decoding](https://mlsanity.com/paper/2104.04784/) (2021)
- [Alternative Visual Units for an Optimized Phoneme-Based Lipreading System](https://mlsanity.com/paper/1909.07147/) (2019)

## Built on by (most cited first, 1 of 1)

- [The Lipreading Gap: Do VSR Models Perceive Visual Speech Like Human Lipreaders?](https://mlsanity.com/paper/2606.07435/) (2026)

## Abstract (arXiv)

This paper presents a novel metric learning approach to address the performance gap between normal and silent speech in visual speech recognition (VSR). The difference in lip movements between the two poses a challenge for existing VSR models, which exhibit degraded accuracy when applied to silent speech. To solve this issue and tackle the scarcity of training data for silent speech, we propose to leverage the shared literal content between normal and silent speech and present a metric learning approach based on visemes. Specifically, we aim to map the input of two speech types close to each other in a latent space if they have similar viseme representations. By minimizing the Kullback-Leibler divergence of the predicted viseme probability distributions between and within the two speech types, our model effectively learns and predicts viseme identities. Our evaluation demonstrates that our method improves the accuracy of silent VSR, even when limited training data is available.