# Improving Speaker-Independent Lipreading with Domain-Adversarial Training

arXiv: https://arxiv.org/abs/1708.01565 · HTML: https://mlsanity.com/paper/1708.01565/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 4 Aug 2017.

## Built on by (most cited first, 7 of 7)

- [Visual Speech Recognition for Multiple Languages in the Wild](https://mlsanity.com/paper/2202.13084/) (2022)
- [LipSound2: Self-Supervised Pre-Training for Lip-to-Speech Reconstruction and Lip Reading](https://mlsanity.com/paper/2112.04748/) (2021)
- [LipFormer: Learning to Lipread Unseen Speakers based on Visual-Landmark Transformers](https://mlsanity.com/paper/2302.02141/) (2023)
- [Landmark-Guided Cross-Speaker Lip Reading with Mutual Information Regularization](https://mlsanity.com/paper/2403.16071/) (2024)
- [EPG2S: Speech Generation and Speech Enhancement based on Electropalatography and Audio Signals using Multimodal Learning](https://mlsanity.com/paper/2206.07860/) (2022)
- [Improving the Gap in Visual Speech Recognition Between Normal and Silent Speech Based on Metric Learning](https://mlsanity.com/paper/2305.14203/) (2023)
- [Exploring Deep Learning for Joint Audio-Visual Lip Biometrics](https://mlsanity.com/paper/2104.08510/) (2021)

## Abstract (arXiv)

We present a Lipreading system, i.e. a speech recognition system using only visual features, which uses domain-adversarial training for speaker independence. Domain-adversarial training is integrated into the optimization of a lipreader based on a stack of feedforward and LSTM (Long Short-Term Memory) recurrent neural networks, yielding an end-to-end trainable system which only requires a very small number of frames of untranscribed target data to substantially improve the recognition accuracy on the target speaker. On pairs of different source and target speakers, we achieve a relative accuracy improvement of around 40% with only 15 to 20 seconds of untranscribed target speech data. On multi-speaker training setups, the accuracy improvements are smaller but still substantial.