# Audio-visual video face hallucination with frequency supervision and cross modality support by speech based lip reading loss

arXiv: https://arxiv.org/abs/2211.10883 · HTML: https://mlsanity.com/paper/2211.10883/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 23 of its 40 references that are in the map.
- First posted on arXiv 20 Nov 2022.

## Builds on (most cited first, 23 of 23)

- [Photo-Realistic Single Image Super-Resolution Using a Generative Adversarial Network](https://mlsanity.com/paper/1609.04802/) (2016)
- [Learning Spatio-Temporal Representation with Pseudo-3D Residual Networks](https://mlsanity.com/paper/1711.10305/) (2017)
- [Frequency Principle: Fourier Analysis Sheds Light on Deep Neural Networks](https://mlsanity.com/paper/1901.06523/) (2019)
- [Towards Real-World Blind Face Restoration with Generative Facial Prior](https://mlsanity.com/paper/2101.04061/) (2021)
- [Deep Audio-Visual Speech Recognition](https://mlsanity.com/paper/1809.02108/) (2018)
- [BasicVSR: The Search for Essential Components in Video Super-Resolution and Beyond](https://mlsanity.com/paper/2012.02181/) (2020)
- [Hierarchical Cross-Modal Talking Face Generationwith Dynamic Pixel-Wise Loss](https://mlsanity.com/paper/1905.03820/) (2019)
- [Focal Frequency Loss for Image Reconstruction and Synthesis](https://mlsanity.com/paper/2012.12821/) (2020)
- [Real-Time Video Super-Resolution with Spatio-Temporal Networks and Motion Compensation](https://mlsanity.com/paper/1611.05250/) (2016)
- [Unified Multisensory Perception: Weakly-Supervised Audio-Visual Video Parsing](https://mlsanity.com/paper/2007.10558/) (2020)
- [Lipreading using Temporal Convolutional Networks](https://mlsanity.com/paper/2001.08702/) (2020)
- [Progressive Face Super-Resolution via Attention to Facial Landmark](https://mlsanity.com/paper/1908.08239/) (2019)
- [Speech2Face: Learning the Face Behind a Voice](https://mlsanity.com/paper/1905.09773/) (2019)
- [LRW-1000: A Naturally-Distributed Large-Scale Benchmark for Lip Reading in the Wild](https://mlsanity.com/paper/1810.06990/) (2018)
- [Super-Identity Convolutional Neural Network for Face Hallucination](https://mlsanity.com/paper/1811.02328/) (2018)
- [GCFSR: a Generative and Controllable Face Super Resolution Method Without Facial and GAN Priors](https://mlsanity.com/paper/2203.07319/) (2022)
- [A survey of face recognition techniques under occlusion](https://mlsanity.com/paper/2006.11366/) (2020)
- [Face Super-Resolution Guided by 3D Facial Priors](https://mlsanity.com/paper/2007.09454/) (2020)
- [Identity-Free Facial Expression Recognition using conditional Generative Adversarial Network](https://mlsanity.com/paper/1903.08051/) (2019)
- [Learning to Have an Ear for Face Super-Resolution](https://mlsanity.com/paper/1909.12780/) (2019)
- [Real-World Super-Resolution of Face-Images from Surveillance Cameras](https://mlsanity.com/paper/2102.03113/) (2021)
- [Face Super-Resolution Through Wasserstein GANs](https://mlsanity.com/paper/1705.02438/) (2017)
- [Frequency Aware Face Hallucination Generative Adversarial Network with Semantic Structural Constraint](https://mlsanity.com/paper/2110.01880/) (2021)

## Abstract (arXiv)

Recently, there has been numerous breakthroughs in face hallucination tasks. However, the task remains rather challenging in videos in comparison to the images due to inherent consistency issues. The presence of extra temporal dimension in video face hallucination makes it non-trivial to learn the facial motion through out the sequence. In order to learn these fine spatio-temporal motion details, we propose a novel cross-modal audio-visual Video Face Hallucination Generative Adversarial Network (VFH-GAN). The architecture exploits the semantic correlation of between the movement of the facial structure and the associated speech signal. Another major issue in present video based approaches is the presence of blurriness around the key facial regions such as mouth and lips - where spatial displacement is much higher in comparison to other areas. The proposed approach explicitly defines a lip reading loss to learn the fine grain motion in these facial areas. During training, GANs have potential to fit frequencies from low to high, which leads to miss the hard to synthesize frequencies. Therefore, to add salient frequency features to the network we add a frequency based loss function. The visual and the quantitative comparison with state-of-the-art shows a significant improvement in performance and efficacy.