# CP-EB: Talking Face Generation with Controllable Pose and Eye Blinking Embedding

arXiv: https://arxiv.org/abs/2311.08673 · HTML: https://mlsanity.com/paper/2311.08673/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 19 of its 33 references that are in the map.
- First posted on arXiv 15 Nov 2023.

## Builds on (most cited first, 19 of 19)

- [Perceptual Losses for Real-Time Style Transfer and Super-Resolution](https://mlsanity.com/paper/1603.08155/) (2016)
- [VoxCeleb2: Deep Speaker Recognition](https://mlsanity.com/paper/1806.05622/) (2018)
- [A Lip Sync Expert Is All You Need for Speech to Lip Generation In The Wild](https://mlsanity.com/paper/2008.10010/) (2020)
- [Deep Speech: Scaling up end-to-end speech recognition](https://mlsanity.com/paper/1412.5567/) (2014)
- [Pose-Controllable Talking Face Generation by Implicitly Modularized Audio-Visual Representation](https://mlsanity.com/paper/2104.11116/) (2021)
- [Motion Representations for Articulated Animation](https://mlsanity.com/paper/2104.11280/) (2021)
- [Hierarchical Cross-Modal Talking Face Generationwith Dynamic Pixel-Wise Loss](https://mlsanity.com/paper/1905.03820/) (2019)
- [Talking Face Generation by Adversarially Disentangled Audio-Visual Representation](https://mlsanity.com/paper/1807.07860/) (2018)
- [EAMM: One-Shot Emotional Talking Face via Audio-Based Emotion-Aware Motion Model](https://mlsanity.com/paper/2205.15278/) (2022)
- [Neural Head Reenactment with Latent Pose Descriptors](https://mlsanity.com/paper/2004.12000/) (2020)
- [FACIAL: Synthesizing Dynamic Talking Face with Implicit Attribute Learning](https://mlsanity.com/paper/2108.07938/) (2021)
- [One-shot Talking Face Generation from Single-speaker Audio-Visual Correlation Learning](https://mlsanity.com/paper/2112.02749/) (2021)
- [Deepfakes Generation and Detection: State-of-the-art, open challenges, countermeasures, and way forward](https://mlsanity.com/paper/2103.00484/) (2021)
- [Head2Head++: Deep Facial Attributes Re-Targeting](https://mlsanity.com/paper/2006.10199/) (2020)
- [StyleMask: Disentangling the Style Space of StyleGAN2 for Neural Face Reenactment](https://mlsanity.com/paper/2209.13375/) (2022)
- [APB2Face: Audio-guided face reenactment with auxiliary pose and blink signals](https://mlsanity.com/paper/2004.14569/) (2020)
- [Pre-Avatar: An Automatic Presentation Generation Framework Leveraging Talking Avatar](https://mlsanity.com/paper/2210.06877/) (2022)
- [Single Source One Shot Reenactment using Weighted motion From Paired Feature Points](https://mlsanity.com/paper/2104.03117/) (2021)
- [Talking Head Generation Driven by Speech-Related Facial Action Units and Audio- Based on Multimodal Representation Fusion](https://mlsanity.com/paper/2204.12756/) (2022)

## Built on by (most cited first, 2 of 2)

- [EmoTalker: Emotionally Editable Talking Face Generation via Diffusion Model](https://mlsanity.com/paper/2401.08049/) (2024)
- [Audio-Driven Talking Face Generation with Blink Embedding and Hash Grid Landmarks Encoding](https://mlsanity.com/paper/2601.18849/) (2026)

## Abstract (arXiv)

This paper proposes a talking face generation method named "CP-EB" that takes an audio signal as input and a person image as reference, to synthesize a photo-realistic people talking video with head poses controlled by a short video clip and proper eye blinking embedding. It's noted that not only the head pose but also eye blinking are both important aspects for deep fake detection. The implicit control of poses by video has already achieved by the state-of-art work. According to recent research, eye blinking has weak correlation with input audio which means eye blinks extraction from audio and generation are possible. Hence, we propose a GAN-based architecture to extract eye blink feature from input audio and reference video respectively and employ contrastive training between them, then embed it into the concatenated features of identity and poses to generate talking face images. Experimental results show that the proposed method can generate photo-realistic talking face with synchronous lips motions, natural head poses and blinking eyes.