# Arbitrary Talking Face Generation via Attentional Audio-Visual Coherence Learning

arXiv: https://arxiv.org/abs/1812.06589 · HTML: https://mlsanity.com/paper/1812.06589/

## Key facts

- Cited by 28 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 17 Dec 2018.

## Built on by (most cited first, 28 of 28)

- [Pose-Controllable Talking Face Generation by Implicitly Modularized Audio-Visual Representation](https://mlsanity.com/paper/2104.11116/) (2021)
- [CelebV-HQ: A Large-Scale Video Facial Attributes Dataset](https://mlsanity.com/paper/2207.12393/) (2022)
- [Audio2Head: Audio-driven One-shot Talking-head Generation with Natural Head Motion](https://mlsanity.com/paper/2107.09293/) (2021)
- [Semantic-Aware Implicit Neural Audio-Driven Video Portrait Generation](https://mlsanity.com/paper/2201.07786/) (2022)
- [Foundations and Trends in Multimodal Machine Learning: Principles, Challenges, and Open Questions](https://mlsanity.com/paper/2209.03430/) (2022)
- [Seeing What You Said: Talking Face Generation Guided by a Lip Reading Expert](https://mlsanity.com/paper/2303.17480/) (2023)
- [Learning Dynamic Facial Radiance Fields for Few-Shot Talking Head Synthesis](https://mlsanity.com/paper/2207.11770/) (2022)
- [Progressive Disentangled Representation Learning for Fine-Grained Controllable Talking Head Synthesis](https://mlsanity.com/paper/2211.14506/) (2022)
- [DFA-NeRF: Personalized Talking Head Generation via Disentangled Face Attributes Neural Rendering](https://mlsanity.com/paper/2201.00791/) (2022)
- [SyncTalkFace: Talking Face Generation with Precise Lip-Syncing via Audio-Lip Memory](https://mlsanity.com/paper/2211.00924/) (2022)
- [RenderMe-360: A Large Digital Asset Library and Benchmarks Towards High-fidelity Head Avatars](https://mlsanity.com/paper/2305.13353/) (2023)
- [Large Language Models are Efficient Learners of Noise-Robust Speech Recognition](https://mlsanity.com/paper/2401.10446/) (2024)
- [MODA: Mapping-Once Audio-driven Portrait Animation with Dual Attentions](https://mlsanity.com/paper/2307.10008/) (2023)
- [Speech Driven Video Editing via an Audio-Conditioned Diffusion Model](https://mlsanity.com/paper/2301.04474/) (2023)
- [Deep Learning for Visual Speech Analysis: A Survey](https://mlsanity.com/paper/2205.10839/) (2022)
- [SingingHead: A Large-scale 4D Dataset for Singing Head Animation](https://mlsanity.com/paper/2312.04369/) (2023)
- [That's What I Said: Fully-Controllable Talking Face Generation](https://mlsanity.com/paper/2304.03275/) (2023)
- [f-Divergence Based Classification: Beyond the Use of Cross-Entropy](https://mlsanity.com/paper/2401.01268/) (2024)
- [Parallel and High-Fidelity Text-to-Lip Generation](https://mlsanity.com/paper/2107.06831/) (2021)
- [Hearing Lips in Noise: Universal Viseme-Phoneme Mapping and Transfer for Robust Audio-Visual Speech Recognition](https://mlsanity.com/paper/2306.10563/) (2023)
- [OSM-Net: One-to-Many One-shot Talking Head Generation with Spontaneous Head Motions](https://mlsanity.com/paper/2309.16148/) (2023)
- [A Survey on Deep Multi-modal Learning for Body Language Recognition and Generation](https://mlsanity.com/paper/2308.08849/) (2023)
- [HDTR-Net: A Real-Time High-Definition Teeth Restoration Network for Arbitrary Talking Face Generation Methods](https://mlsanity.com/paper/2309.07495/) (2023)
- [A Keypoint Based Enhancement Method for Audio Driven Free View Talking Head Synthesis](https://mlsanity.com/paper/2210.03335/) (2022)
- [A Comprehensive Multi-scale Approach for Speech and Dynamics Synchrony in Talking Head Generation](https://mlsanity.com/paper/2307.03270/) (2023)
- [G4G:A Generic Framework for High Fidelity Talking Face Generation with Fine-grained Intra-modal Alignment](https://mlsanity.com/paper/2402.18122/) (2024)
- [DEMO: Disentangled Motion Latent Flow Matching for Fine-Grained Controllable Talking Portrait Synthesis](https://mlsanity.com/paper/2510.10650/) (2025)
- [GaussianEmoTalker: Real-Time Emotional Talking Head Synthesis with Audio-Driven and Blendshape-Based 3D Gaussian Splatting](https://mlsanity.com/paper/2607.00959/) (2026)

## Abstract (arXiv)

Talking face generation aims to synthesize a face video with precise lip synchronization as well as a smooth transition of facial motion over the entire video via the given speech clip and facial image. Most existing methods mainly focus on either disentangling the information in a single image or learning temporal information between frames. However, cross-modality coherence between audio and video information has not been well addressed during synthesis. In this paper, we propose a novel arbitrary talking face generation framework by discovering the audio-visual coherence via the proposed Asymmetric Mutual Information Estimator (AMIE). In addition, we propose a Dynamic Attention (DA) block by selectively focusing the lip area of the input image during the training stage, to further enhance lip synchronization. Experimental results on benchmark LRW dataset and GRID dataset transcend the state-of-the-art methods on prevalent metrics with robust high-resolution synthesizing on gender and pose variations.