# VisageSynTalk: Unseen Speaker Video-to-Speech Synthesis via Speech-Visage Feature Selection

arXiv: https://arxiv.org/abs/2206.07458 · HTML: https://mlsanity.com/paper/2206.07458/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- Builds on 27 of its 47 references that are in the map.
- First posted on arXiv 15 Jun 2022.

## Builds on (most cited first, 27 of 27)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Grad-CAM: Visual Explanations from Deep Networks via Gradient-based Localization](https://mlsanity.com/paper/1610.02391/) (2016)
- [A Style-Based Generator Architecture for Generative Adversarial Networks](https://mlsanity.com/paper/1812.04948/) (2018)
- [Conditional Generative Adversarial Nets](https://mlsanity.com/paper/1411.1784/) (2014)
- [Arbitrary Style Transfer in Real-time with Adaptive Instance Normalization](https://mlsanity.com/paper/1703.06868/) (2017)
- [Unsupervised Domain Adaptation by Backpropagation](https://mlsanity.com/paper/1409.7495/) (2014)
- [Multimodal Unsupervised Image-to-Image Translation](https://mlsanity.com/paper/1804.04732/) (2018)
- [VoxCeleb: a large-scale speaker identification dataset](https://mlsanity.com/paper/1706.08612/) (2017)
- [MS-Celeb-1M: A Dataset and Benchmark for Large-Scale Face Recognition](https://mlsanity.com/paper/1607.08221/) (2016)
- [Tacotron: Towards End-to-End Speech Synthesis](https://mlsanity.com/paper/1703.10135/) (2017)
- [S3FD: Single Shot Scale-invariant Face Detector](https://mlsanity.com/paper/1708.05237/) (2017)
- [LipNet: End-to-End Sentence-level Lipreading](https://mlsanity.com/paper/1611.01599/) (2016)
- [Learning Individual Speaking Styles for Accurate Lip to Speech Synthesis](https://mlsanity.com/paper/2005.08209/) (2020)
- [Combining Residual Networks with LSTMs for Lipreading](https://mlsanity.com/paper/1703.04105/) (2017)
- [Vid2speech: Speech Reconstruction from Silent Video](https://mlsanity.com/paper/1701.00495/) (2017)
- [AGAIN-VC: A One-shot Voice Conversion using Activation Guidance and Adaptive Instance Normalization](https://mlsanity.com/paper/2011.00316/) (2020)
- [Lip2AudSpec: Speech reconstruction from silent lip movements video](https://mlsanity.com/paper/1710.09798/) (2017)
- [End-to-End Speech-Driven Facial Animation with Temporal GANs](https://mlsanity.com/paper/1805.09313/) (2018)
- [End-to-End Video-To-Speech Synthesis using Generative Adversarial Networks](https://mlsanity.com/paper/2104.13332/) (2021)
- [Improved Speech Reconstruction from Silent Video](https://mlsanity.com/paper/1708.01204/) (2017)
- [Video-Driven Speech Reconstruction using Generative Adversarial Networks](https://mlsanity.com/paper/1906.06301/) (2019)
- [AFS: An Attention-based mechanism for Supervised Feature Selection](https://mlsanity.com/paper/1902.11074/) (2019)
- [Vocoder-Based Speech Synthesis from Silent Videos](https://mlsanity.com/paper/2004.02541/) (2020)
- [Speech Prediction in Silent Videos using Variational Autoencoders](https://mlsanity.com/paper/2011.07340/) (2020)
- [Multi-modality Associative Bridging through Memory: Speech Sound Recollected from Face Video](https://mlsanity.com/paper/2204.01265/) (2022)
- [Feature Selection Using Batch-Wise Attenuation and Feature Mask Normalization](https://mlsanity.com/paper/2010.13631/) (2020)

## Built on by (most cited first, 6 of 6)

- [Watch or Listen: Robust Audio-Visual Speech Recognition with Visual Corruption Modeling and Reliability Scoring](https://mlsanity.com/paper/2303.08536/) (2023)
- [DiffV2S: Diffusion-based Video-to-Speech Synthesis with Vision-guided Speaker Embedding](https://mlsanity.com/paper/2308.07787/) (2023)
- [Text-driven Talking Face Synthesis by Reprogramming Audio-driven Models](https://mlsanity.com/paper/2306.16003/) (2023)
- [EgoSonics: Generating Synchronized Audio for Silent Egocentric Videos](https://mlsanity.com/paper/2407.20592/) (2024)
- [Large-scale unsupervised audio pre-training for video-to-speech synthesis](https://mlsanity.com/paper/2306.15464/) (2023)
- [NaturalL2S: End-to-End High-quality Multispeaker Lip-to-Speech Synthesis with Differential Digital Signal Processing](https://mlsanity.com/paper/2502.12002/) (2025)

## Abstract (arXiv)

The goal of this work is to reconstruct speech from a silent talking face video. Recent studies have shown impressive performance on synthesizing speech from silent talking face videos. However, they have not explicitly considered on varying identity characteristics of different speakers, which place a challenge in the video-to-speech synthesis, and this becomes more critical in unseen-speaker settings. Our approach is to separate the speech content and the visage-style from a given silent talking face video. By guiding the model to independently focus on modeling the two representations, we can obtain the speech of high intelligibility from the model even when the input video of an unseen subject is given. To this end, we introduce speech-visage selection that separates the speech content and the speaker identity from the visual features of the input video. The disentangled representations are jointly incorporated to synthesize speech through visage-style based synthesizer which generates speech by coating the visage-styles while maintaining the speech content. Thus, the proposed framework brings the advantage of synthesizing the speech containing the right content even with the silent talking face video of an unseen subject. We validate the effectiveness of the proposed framework on the GRID, TCD-TIMIT volunteer, and LRW datasets.