# Leveraging Visemes for Better Visual Speech Representation and Lip Reading

arXiv: https://arxiv.org/abs/2307.10157 · HTML: https://mlsanity.com/paper/2307.10157/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 19 of its 20 references that are in the map.
- First posted on arXiv 19 Jul 2023.

## Builds on (most cited first, 19 of 19)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Very Deep Convolutional Networks for Large-Scale Image Recognition](https://mlsanity.com/paper/1409.1556/) (2014)
- [Conformer: Convolution-augmented Transformer for Speech Recognition](https://mlsanity.com/paper/2005.08100/) (2020)
- [Common Voice: A Massively-Multilingual Speech Corpus](https://mlsanity.com/paper/1912.06670/) (2019)
- [VoxCeleb2: Deep Speaker Recognition](https://mlsanity.com/paper/1806.05622/) (2018)
- [Deep Audio-Visual Speech Recognition](https://mlsanity.com/paper/1809.02108/) (2018)
- [LRS3-TED: a large-scale dataset for visual speech recognition](https://mlsanity.com/paper/1809.00496/) (2018)
- [Learning Audio-Visual Speech Representation by Masked Multimodal Cluster Prediction](https://mlsanity.com/paper/2201.02184/) (2022)
- [Lip Reading Sentences in the Wild](https://mlsanity.com/paper/1611.05358/) (2016)
- [End-to-end Audio-visual Speech Recognition with Conformers](https://mlsanity.com/paper/2102.06657/) (2021)
- [ASR is all you need: cross-modal distillation for lip reading](https://mlsanity.com/paper/1911.12747/) (2019)
- [Audio-Visual Speech Recognition With A Hybrid CTC/Attention Architecture](https://mlsanity.com/paper/1810.00108/) (2018)
- [LRW-1000: A Naturally-Distributed Large-Scale Benchmark for Lip Reading in the Wild](https://mlsanity.com/paper/1810.06990/) (2018)
- [Jointly Learning Visual and Auditory Speech Representations from Raw Data](https://mlsanity.com/paper/2212.06246/) (2022)
- [Audio-visual Recognition of Overlapped speech for the LRS2 dataset](https://mlsanity.com/paper/2001.01656/) (2020)
- [Decoding visemes: improving machine lipreading](https://mlsanity.com/paper/1710.01288/) (2017)
- [Lip reading using external viseme decoding](https://mlsanity.com/paper/2104.04784/) (2021)
- [A Multi-Purpose Audio-Visual Corpus for Multi-Modal Persian Speech Recognition: the Arman-AV Dataset](https://mlsanity.com/paper/2301.10180/) (2023)
- [Word-level Persian Lipreading Dataset](https://mlsanity.com/paper/2304.04068/) (2023)

## Built on by (most cited first, 2 of 2)

- [VALLR: Visual ASR Language Model for Lip Reading](https://mlsanity.com/paper/2503.21408/) (2025)
- [Phoneme-Level Visual Speech Recognition via Point-Visual Fusion and Language Model Reconstruction](https://mlsanity.com/paper/2507.18863/) (2025)

## Abstract (arXiv)

Lip reading is a challenging task that has many potential applications in speech recognition, human-computer interaction, and security systems. However, existing lip reading systems often suffer from low accuracy due to the limitations of video features. In this paper, we propose a novel approach that leverages visemes, which are groups of phonetically similar lip shapes, to extract more discriminative and robust video features for lip reading. We evaluate our approach on various tasks, including word-level and sentence-level lip reading, and audiovisual speech recognition using the Arman-AV dataset, a largescale Persian corpus. Our experimental results show that our viseme based approach consistently outperforms the state-of-theart methods in all these tasks. The proposed method reduces the lip-reading word error rate (WER) by 9.1% relative to the best previous method.