# Attention-Based Lip Audio-Visual Synthesis for Talking Face Generation in the Wild

arXiv: https://arxiv.org/abs/2203.03984 · HTML: https://mlsanity.com/paper/2203.03984/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- Builds on 23 of its 36 references that are in the map.
- First posted on arXiv 8 Mar 2022.

## Builds on (most cited first, 23 of 23)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Generative Adversarial Networks](https://mlsanity.com/paper/2203.00667/) (2022)
- [Neural Machine Translation by Jointly Learning to Align and Translate](https://mlsanity.com/paper/1409.0473/) (2014)
- [Image-to-Image Translation with Conditional Adversarial Networks](https://mlsanity.com/paper/1611.07004/) (2016)
- [Progressive Growing of GANs for Improved Quality, Stability, and Variation](https://mlsanity.com/paper/1710.10196/) (2017)
- [CBAM: Convolutional Block Attention Module](https://mlsanity.com/paper/1807.06521/) (2018)
- [Conditional Generative Adversarial Nets](https://mlsanity.com/paper/1411.1784/) (2014)
- [Unsupervised Representation Learning with Deep Convolutional Generative Adversarial Networks](https://mlsanity.com/paper/1511.06434/) (2015)
- [Spatial Transformer Networks](https://mlsanity.com/paper/1506.02025/) (2015)
- [Show, Attend and Tell: Neural Image Caption Generation with Visual Attention](https://mlsanity.com/paper/1502.03044/) (2015)
- [Effective Approaches to Attention-based Neural Machine Translation](https://mlsanity.com/paper/1508.04025/) (2015)
- [A Lip Sync Expert Is All You Need for Speech to Lip Generation In The Wild](https://mlsanity.com/paper/2008.10010/) (2020)
- [StackGAN: Text to Photo-realistic Image Synthesis with Stacked Generative Adversarial Networks](https://mlsanity.com/paper/1612.03242/) (2016)
- [Recurrent Models of Visual Attention](https://mlsanity.com/paper/1406.6247/) (2014)
- [Deep Audio-Visual Speech Recognition](https://mlsanity.com/paper/1809.02108/) (2018)
- [LRS3-TED: a large-scale dataset for visual speech recognition](https://mlsanity.com/paper/1809.00496/) (2018)
- [Talking Face Generation by Adversarially Disentangled Audio-Visual Representation](https://mlsanity.com/paper/1807.07860/) (2018)
- [S3FD: Single Shot Scale-invariant Face Detector](https://mlsanity.com/paper/1708.05237/) (2017)
- [You said that?](https://mlsanity.com/paper/1705.02966/) (2017)
- [Towards Automatic Face-to-Face Translation](https://mlsanity.com/paper/2003.00418/) (2020)
- [ObamaNet: Photo-realistic lip-sync from text](https://mlsanity.com/paper/1801.01442/) (2017)
- [ABCNN: Attention-Based Convolutional Neural Network for Modeling Sentence Pairs](https://mlsanity.com/paper/1512.05193/) (2015)

## Built on by (most cited first, 7 of 7)

- [MARLIN: Masked Autoencoder for facial video Representation LearnINg](https://mlsanity.com/paper/2211.06627/) (2022)
- [Emotionally Enhanced Talking Face Generation](https://mlsanity.com/paper/2303.11548/) (2023)
- [An Audio-Visual Attention Based Multimodal Network for Fake Talking Face Videos Detection](https://mlsanity.com/paper/2203.05178/) (2022)
- [HDTR-Net: A Real-Time High-Definition Teeth Restoration Network for Arbitrary Talking Face Generation Methods](https://mlsanity.com/paper/2309.07495/) (2023)
- [FTFDNet: Learning to Detect Talking Face Video Manipulation with Tri-Modality Interaction](https://mlsanity.com/paper/2307.03990/) (2023)
- [Lightweight Joint Audio-Visual Deepfake Detection via Single-Stream Multi-Modal Learning Framework](https://mlsanity.com/paper/2506.07358/) (2025)
- [LPIPS-AttnWav2Lip: Generic Audio-Driven lip synchronization for Talking Head Generation in the Wild](https://mlsanity.com/paper/2602.00189/) (2026)

## Abstract (arXiv)

Talking face generation with great practical significance has attracted more attention in recent audio-visual studies. How to achieve accurate lip synchronization is a long-standing challenge to be further investigated. Motivated by xxx, in this paper, an AttnWav2Lip model is proposed by incorporating spatial attention module and channel attention module into lip-syncing strategy. Rather than focusing on the unimportant regions of the face image, the proposed AttnWav2Lip model is able to pay more attention on the lip region reconstruction. To our limited knowledge, this is the first attempt to introduce attention mechanism to the scheme of talking face generation. An extensive experiments have been conducted to evaluate the effectiveness of the proposed model. Compared to the baseline measured by LSE-D and LSE-C metrics, a superior performance has been demonstrated on the benchmark lip synthesis datasets, including LRW, LRS2 and LRS3.