# Learning Speaker-Invariant Visual Features for Lipreading

arXiv: https://arxiv.org/abs/2506.07572 · HTML: https://mlsanity.com/paper/2506.07572/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 36 of its 66 references that are in the map.
- First posted on arXiv 9 Jun 2025.

## Builds on (most cited first, 30 of 36)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Robust Speech Recognition via Large-Scale Weak Supervision](https://mlsanity.com/paper/2212.04356/) (2022)
- [Domain-Adversarial Training of Neural Networks](https://mlsanity.com/paper/1505.07818/) (2015)
- [Sequence to Sequence Learning with Neural Networks](https://mlsanity.com/paper/1409.3215/) (2014)
- [Exploring CLIP for Assessing the Look and Feel of Images](https://mlsanity.com/paper/2207.12396/) (2022)
- [CLIP4Clip: An Empirical Study of CLIP for End to End Video Clip Retrieval](https://mlsanity.com/paper/2104.08860/) (2021)
- [Deep Audio-Visual Speech Recognition](https://mlsanity.com/paper/1809.02108/) (2018)
- [Lip Reading Sentences in the Wild](https://mlsanity.com/paper/1611.05358/) (2016)
- [CLIP2Video: Mastering Video-Text Retrieval via Image CLIP](https://mlsanity.com/paper/2106.11097/) (2021)
- [Gloss-free Sign Language Translation: Improving from Visual-Language Pretraining](https://mlsanity.com/paper/2307.14768/) (2023)
- [Lipreading using Temporal Convolutional Networks](https://mlsanity.com/paper/2001.08702/) (2020)
- [ASR is all you need: cross-modal distillation for lip reading](https://mlsanity.com/paper/1911.12747/) (2019)
- [LipNet: End-to-End Sentence-level Lipreading](https://mlsanity.com/paper/1611.01599/) (2016)
- [Video-based Remote Physiological Measurement via Cross-verified Feature Disentangling](https://mlsanity.com/paper/2007.08213/) (2020)
- [End-to-end Audiovisual Speech Recognition](https://mlsanity.com/paper/1802.06424/) (2018)
- [Pose-guided Feature Disentangling for Occluded Person Re-identification Based on Transformer](https://mlsanity.com/paper/2112.02466/) (2021)
- [Audio-Visual Speech Recognition With A Hybrid CTC/Attention Architecture](https://mlsanity.com/paper/1810.00108/) (2018)
- [Sub-word Level Lip Reading With Visual Attention](https://mlsanity.com/paper/2110.07603/) (2021)
- [Discriminative Multi-modality Speech Recognition](https://mlsanity.com/paper/2005.05592/) (2020)
- [Towards Practical Lipreading with Distilled and Efficient Models](https://mlsanity.com/paper/2007.06504/) (2020)
- [Hearing Lips: Improving Lip Reading by Distilling Speech Recognizers](https://mlsanity.com/paper/1911.11502/) (2019)
- [Audio-visual Recognition of Overlapped speech for the LRS2 dataset](https://mlsanity.com/paper/2001.01656/) (2020)
- [CiCo: Domain-Aware Sign Language Retrieval via Cross-Lingual Contrastive Learning](https://mlsanity.com/paper/2303.12793/) (2023)
- [Variational Feature Disentangling for Fine-Grained Few-Shot Classification](https://mlsanity.com/paper/2010.03255/) (2020)
- [A Cascade Sequence-to-Sequence Model for Chinese Mandarin Lip Reading](https://mlsanity.com/paper/1908.04917/) (2019)
- [Lipper: Synthesizing Thy Speech using Multi-View Lipreading](https://mlsanity.com/paper/1907.01367/) (2019)
- [Speaker-adaptive Lip Reading with User-dependent Padding](https://mlsanity.com/paper/2208.04498/) (2022)
- [Lip Reading for Low-resource Languages by Learning and Combining General Speech Knowledge and Language-specific Knowledge](https://mlsanity.com/paper/2308.09311/) (2023)
- [LCANet: End-to-End Lipreading with Cascaded Attention-CTC](https://mlsanity.com/paper/1803.04988/) (2018)
- [Decompose to Adapt: Cross-domain Object Detection via Feature Disentanglement](https://mlsanity.com/paper/2201.01929/) (2022)

## Abstract (arXiv)

Lipreading is a challenging cross-modal task that aims to convert visual lip movements into spoken text. Existing lipreading methods often extract visual features that include speaker-specific lip attributes (e.g., shape, color, texture), which introduce spurious correlations between vision and text. These correlations lead to suboptimal lipreading accuracy and restrict model generalization. To address this challenge, we introduce SIFLip, a speaker-invariant visual feature learning framework that disentangles speaker-specific attributes using two complementary disentanglement modules (Implicit Disentanglement and Explicit Disentanglement) to improve generalization. Specifically, since different speakers exhibit semantic consistency between lip movements and phonetic text when pronouncing the same words, our implicit disentanglement module leverages stable text embeddings as supervisory signals to learn common visual representations across speakers, implicitly decoupling speaker-specific features. Additionally, we design a speaker recognition sub-task within the main lipreading pipeline to filter speaker-specific features, then further explicitly disentangle these personalized visual features from the backbone network via gradient reversal. Experimental results demonstrate that SIFLip significantly enhances generalization performance across multiple public datasets. Experimental results demonstrate that SIFLip significantly improves generalization performance across multiple public datasets, outperforming state-of-the-art methods.