# Lipper: Synthesizing Thy Speech using Multi-View Lipreading

arXiv: https://arxiv.org/abs/1907.01367 · HTML: https://mlsanity.com/paper/1907.01367/

## Key facts

- Cited by 19 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 28 Jun 2019.

## Built on by (most cited first, 19 of 19)

- [Neural Dubber: Dubbing for Videos According to Scripts](https://mlsanity.com/paper/2110.08243/) (2021)
- [Learning to Dub Movies via Hierarchical Prosody Models](https://mlsanity.com/paper/2212.04054/) (2022)
- [End-to-End Video-To-Speech Synthesis using Generative Adversarial Networks](https://mlsanity.com/paper/2104.13332/) (2021)
- [Audio-Visual Speech Codecs: Rethinking Audio-Visual Speech Enhancement by Re-Synthesis](https://mlsanity.com/paper/2203.17263/) (2022)
- [More than Words: In-the-Wild Visually-Driven Prosody for Text-to-Speech](https://mlsanity.com/paper/2111.10139/) (2021)
- [Let There Be Sound: Reconstructing High Quality Speech from Silent Videos](https://mlsanity.com/paper/2308.15256/) (2023)
- [LipSound2: Self-Supervised Pre-Training for Lip-to-Speech Reconstruction and Lip Reading](https://mlsanity.com/paper/2112.04748/) (2021)
- [From Faces to Voices: Learning Hierarchical Representations for High-quality Video-to-Speech](https://mlsanity.com/paper/2503.16956/) (2025)
- [Lip-to-Speech Synthesis for Arbitrary Speakers in the Wild](https://mlsanity.com/paper/2209.00642/) (2022)
- [AlignDiT: Multimodal Aligned Diffusion Transformer for Synchronized Speech Generation](https://mlsanity.com/paper/2504.20629/) (2025)
- [VCVTS: Multi-speaker Video-to-Speech synthesis via cross-modal knowledge transfer from voice conversion](https://mlsanity.com/paper/2202.09081/) (2022)
- [V2SFlow: Video-to-Speech Generation with Speech Decomposition and Rectified Flow](https://mlsanity.com/paper/2411.19486/) (2024)
- [Facetron: A Multi-speaker Face-to-Speech Model based on Cross-modal Latent Representations](https://mlsanity.com/paper/2107.12003/) (2021)
- [A Comprehensive Survey on Generative AI for Video-to-Music Generation](https://mlsanity.com/paper/2502.12489/) (2025)
- [Multi-Perspective LSTM for Joint Visual Representation Learning](https://mlsanity.com/paper/2105.02802/) (2021)
- [Learning Speaker-specific Lip-to-Speech Generation](https://mlsanity.com/paper/2206.02050/) (2022)
- [NaturalL2S: End-to-End High-quality Multispeaker Lip-to-Speech Synthesis with Differential Digital Signal Processing](https://mlsanity.com/paper/2502.12002/) (2025)
- [Show Me Your Face, And I'll Tell You How You Speak](https://mlsanity.com/paper/2206.14009/) (2022)
- [Learning Speaker-Invariant Visual Features for Lipreading](https://mlsanity.com/paper/2506.07572/) (2025)

## Abstract (arXiv)

Lipreading has a lot of potential applications such as in the domain of surveillance and video conferencing. Despite this, most of the work in building lipreading systems has been limited to classifying silent videos into classes representing text phrases. However, there are multiple problems associated with making lipreading a text-based classification task like its dependence on a particular language and vocabulary mapping. Thus, in this paper we propose a multi-view lipreading to audio system, namely Lipper, which models it as a regression task. The model takes silent videos as input and produces speech as the output. With multi-view silent videos, we observe an improvement over single-view speech reconstruction results. We show this by presenting an exhaustive set of experiments for speaker-dependent, out-of-vocabulary and speaker-independent settings. Further, we compare the delay values of Lipper with other speechreading systems in order to show the real-time nature of audio produced. We also perform a user study for the audios produced in order to understand the level of comprehensibility of audios produced using Lipper.