# APB2Face: Audio-guided face reenactment with auxiliary pose and blink signals

arXiv: https://arxiv.org/abs/2004.14569 · HTML: https://mlsanity.com/paper/2004.14569/

## Key facts

- Cited by 14 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 30 Apr 2020.

## Built on by (most cited first, 14 of 14)

- [Identity-Preserving Talking Face Generation with Landmark and Appearance Priors](https://mlsanity.com/paper/2305.08293/) (2023)
- [From Audio to Photoreal Embodiment: Synthesizing Humans in Conversations](https://mlsanity.com/paper/2401.01885/) (2024)
- [Deepfake Generation and Detection: A Benchmark and Survey](https://mlsanity.com/paper/2403.17881/) (2024)
- [Talking Head Generation with Probabilistic Audio-to-Visual Diffusion Priors](https://mlsanity.com/paper/2212.04248/) (2022)
- [High-fidelity Generalized Emotional Talking Face Generation with Multi-modal Emotion Space Learning](https://mlsanity.com/paper/2305.02572/) (2023)
- [A Generalist FaceX via Learning Unified Facial Representation](https://mlsanity.com/paper/2401.00551/) (2023)
- [Phasic Content Fusing Diffusion Model with Directional Distribution Consistency for Few-Shot Model Adaption](https://mlsanity.com/paper/2309.03729/) (2023)
- [DisCoHead: Audio-and-Video-Driven Talking Head Generation by Disentangled Control of Head Pose and Facial Expressions](https://mlsanity.com/paper/2303.07697/) (2023)
- [DyaDiT: A Multi-Modal Diffusion Transformer for Socially Favorable Dyadic Gesture Generation](https://mlsanity.com/paper/2602.23165/) (2026)
- [Video Manipulations Beyond Faces: A Dataset with Human-Machine Analysis](https://mlsanity.com/paper/2207.13064/) (2022)
- [MMoFusion: Multi-modal Co-Speech Motion Generation with Diffusion Model](https://mlsanity.com/paper/2403.02905/) (2024)
- [CP-EB: Talking Face Generation with Controllable Pose and Eye Blinking Embedding](https://mlsanity.com/paper/2311.08673/) (2023)
- [MIMAFace: Face Animation via Motion-Identity Modulated Appearance Feature Learning](https://mlsanity.com/paper/2409.15179/) (2024)
- [Identity-guided Face Generation with Multi-modal Contour Conditions](https://mlsanity.com/paper/2110.04854/) (2021)

## Abstract (arXiv)

Audio-guided face reenactment aims at generating photorealistic faces using audio information while maintaining the same facial movement as when speaking to a real person. However, existing methods can not generate vivid face images or only reenact low-resolution faces, which limits the application value. To solve those problems, we propose a novel deep neural network named APB2Face, which consists of GeometryPredictor and FaceReenactor modules. GeometryPredictor uses extra head pose and blink state signals as well as audio to predict the latent landmark geometry information, while FaceReenactor inputs the face landmark image to reenact the photorealistic face. A new dataset AnnVI collected from YouTube is presented to support the approach, and experimental results indicate the superiority of our method than state-of-the-arts, whether in authenticity or controllability.