# LipFormer: Learning to Lipread Unseen Speakers based on Visual-Landmark Transformers

arXiv: https://arxiv.org/abs/2302.02141 · HTML: https://mlsanity.com/paper/2302.02141/

## Key facts

- Cited by 9 papers in the map (data through 14 Oct 2026).
- Builds on 23 of its 51 references that are in the map.
- First posted on arXiv 4 Feb 2023.

## Builds on (most cited first, 23 of 23)

- [Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modeling](https://mlsanity.com/paper/1412.3555/) (2014)
- [Deep Audio-Visual Speech Recognition](https://mlsanity.com/paper/1809.02108/) (2018)
- [Lip Reading Sentences in the Wild](https://mlsanity.com/paper/1611.05358/) (2016)
- [Return of the Devil in the Details: Delving Deep into Convolutional Nets](https://mlsanity.com/paper/1405.3531/) (2014)
- [Visual Speech Recognition for Multiple Languages in the Wild](https://mlsanity.com/paper/2202.13084/) (2022)
- [LipNet: End-to-End Sentence-level Lipreading](https://mlsanity.com/paper/1611.01599/) (2016)
- [End-to-end Audiovisual Speech Recognition](https://mlsanity.com/paper/1802.06424/) (2018)
- [Unsupervised Discovery of Object Landmarks as Structural Representations](https://mlsanity.com/paper/1804.04412/) (2018)
- [Combining Residual Networks with LSTMs for Lipreading](https://mlsanity.com/paper/1703.04105/) (2017)
- [Sub-word Level Lip Reading With Visual Attention](https://mlsanity.com/paper/2110.07603/) (2021)
- [Discriminative Multi-modality Speech Recognition](https://mlsanity.com/paper/2005.05592/) (2020)
- [CTNet: Context-based Tandem Network for Semantic Segmentation](https://mlsanity.com/paper/2104.09805/) (2021)
- [Hearing Lips: Improving Lip Reading by Distilling Speech Recognizers](https://mlsanity.com/paper/1911.11502/) (2019)
- [A Cascade Sequence-to-Sequence Model for Chinese Mandarin Lip Reading](https://mlsanity.com/paper/1908.04917/) (2019)
- [Syllable-Based Sequence-to-Sequence Speech Recognition with the Transformer in Mandarin Chinese](https://mlsanity.com/paper/1804.10752/) (2018)
- [Mutual Information Maximization for Effective Lip Reading](https://mlsanity.com/paper/2003.06439/) (2020)
- [LCANet: End-to-End Lipreading with Cascaded Attention-CTC](https://mlsanity.com/paper/1803.04988/) (2018)
- [3D PersonVLAD: Learning Deep Global Representations for Video-based Person Re-identification](https://mlsanity.com/paper/1812.10222/) (2018)
- [Improving Speaker-Independent Lipreading with Domain-Adversarial Training](https://mlsanity.com/paper/1708.01565/) (2017)
- [Deep Co-attention based Comparators For Relative Representation Learning in Person Re-identification](https://mlsanity.com/paper/1804.11027/) (2018)
- [FastLR: Non-Autoregressive Lipreading Model with Integrate-and-Fire](https://mlsanity.com/paper/2008.02516/) (2020)
- [Investigations on End-to-End Audiovisual Fusion](https://mlsanity.com/paper/1804.11127/) (2018)
- [Real-Time Facial Expression Recognition using Facial Landmarks and Neural Networks](https://mlsanity.com/paper/2202.00102/) (2022)

## Built on by (most cited first, 9 of 9)

- [CorrTalk: Correlation Between Hierarchical Speech and Facial Activity Variances for 3D Animation](https://mlsanity.com/paper/2310.11295/) (2023)
- [A Novel Approach to Industrial Defect Generation through Blended Latent Diffusion Model with Online Adaptation](https://mlsanity.com/paper/2402.19330/) (2024)
- [Landmark-Guided Cross-Speaker Lip Reading with Mutual Information Regularization](https://mlsanity.com/paper/2403.16071/) (2024)
- [UTSRMorph: A Unified Transformer and Superresolution Network for Unsupervised Medical Image Registration](https://mlsanity.com/paper/2410.20348/) (2024)
- [SignBind-LLM: Multi-Stage Modality Fusion for Sign Language Translation](https://mlsanity.com/paper/2509.00030/) (2025)
- [Automated Speaker Independent Visual Speech Recognition: A Comprehensive Survey](https://mlsanity.com/paper/2306.08314/) (2023)
- [Learning Speaker-Invariant Visual Features for Lipreading](https://mlsanity.com/paper/2506.07572/) (2025)
- [Landmark Guided Visual Feature Extractor for Visual Speech Recognition with Limited Resource](https://mlsanity.com/paper/2508.07233/) (2025)
- [Cascade-Free Mandarin Visual Speech Recognition via Semantic-Guided Cross-Representation Alignment](https://mlsanity.com/paper/2603.21808/) (2026)

## Abstract (arXiv)

Lipreading refers to understanding and further translating the speech of a speaker in the video into natural language. State-of-the-art lipreading methods excel in interpreting overlap speakers, i.e., speakers appear in both training and inference sets. However, generalizing these methods to unseen speakers incurs catastrophic performance degradation due to the limited number of speakers in training bank and the evident visual variations caused by the shape/color of lips for different speakers. Therefore, merely depending on the visible changes of lips tends to cause model overfitting. To address this problem, we propose to use multi-modal features across visual and landmarks, which can describe the lip motion irrespective to the speaker identities. Then, we develop a sentence-level lipreading framework based on visual-landmark transformers, namely LipFormer. Specifically, LipFormer consists of a lip motion stream, a facial landmark stream, and a cross-modal fusion. The embeddings from the two streams are produced by self-attention, which are fed to the cross-attention module to achieve the alignment between visuals and landmarks. Finally, the resulting fused features can be decoded to output texts by a cascade seq2seq model. Experiments demonstrate that our method can effectively enhance the model generalization to unseen speakers.