mlsanityData to 14 Oct 2026

Emotional Talking Head Generation based on Memory-Sharing and Attention-Augmented Networks

arXiv 2306.03594 ↗First posted 6 Jun 2023cs.CVcited by 7

Not on the map. Why?

Abstract (arXiv)

Given an audio clip and a reference face image, the goal of the talking head generation is to generate a high-fidelity talking head video. Although some audio-driven methods of generating talking head videos have made some achievements in the past, most of them only focused on lip and audio synchronization and lack the ability to reproduce the facial expressions of the target person. To this end, we propose a talking head generation model consisting of a Memory-Sharing Emotion Feature extractor (MSEF) and an Attention-Augmented Translator based on U-net (AATU). Firstly, MSEF can extract implicit emotional auxiliary features from audio to estimate more accurate emotional face landmarks.~Secondly, AATU acts as a translator between the estimated landmarks and the photo-realistic video frames. Extensive qualitative and quantitative experiments have shown the superiority of the proposed method to the previous works. Codes will be made publicly available.

Key facts. Cited by 7 papers in the map (data through 14 Oct 2026). Builds on 28 of its 36 references that are in the map. First posted on arXiv 6 Jun 2023.

Builds on

28 of 36 references.

References · 28

  1. 2014Very Deep Convolutional Networks for Large-Scale Image Recognitioncited by 15,364
  2. 2016Perceptual Losses for Real-Time Style Transfer and Super-Resolutioncited by 2,694
  3. 2018CBAM: Convolutional Block Attention Modulecited by 2,477
  4. 2017VoxCeleb: a large-scale speaker identification datasetcited by 732
  5. 2020A Lip Sync Expert Is All You Need for Speech to Lip Generation In The Wildcited by 672
  6. 2020One-Shot Free-View Neural Talking-Head Synthesis for Video Conferencingcited by 444
  7. 2021AD-NeRF: Audio Driven Neural Radiance Fields for Talking Head Synthesiscited by 339
  8. 2020MakeItTalk: Speaker-Aware Talking-Head Animationcited by 312
  9. 2021Pose-Controllable Talking Face Generation by Implicitly Modularized Audio-Visual Representationcited by 288
  10. 2019Hierarchical Cross-Modal Talking Face Generationwith Dynamic Pixel-Wise Losscited by 252
  11. 2019Neural Voice Puppetry: Audio-driven Facial Reenactmentcited by 250
  12. 2021Audio-Driven Emotional Video Portraitscited by 180
16 more
  1. 2022EAMM: One-Shot Emotional Talking Face via Audio-Based Emotion-Aware Motion Modelcited by 151
  2. 2021Audio2Head: Audio-driven One-shot Talking-head Generation with Natural Head Motioncited by 149
  3. 2019Realistic Speech-Driven Facial Animation with GANscited by 142
  4. 2021Beyond Self-attention: External Attention using Two Linear Layers for Visual Taskscited by 109
  5. 2021FACIAL: Synthesizing Dynamic Talking Face with Implicit Attribute Learningcited by 90
  6. 2020Everybody's Talkin': Let Me Talk as You Wantcited by 83
  7. 2018LRW-1000: A Naturally-Distributed Large-Scale Benchmark for Lip Reading in the Wildcited by 51
  8. 2019Neural Style-Preserving Visual Dubbingcited by 47
  9. 2020Audio- and Gaze-driven Facial Animation of Codec Avatarscited by 46
  10. 2022Audio-Driven Talking Face Video Generation with Dynamic Convolution Kernelscited by 23
  11. 2018Speech-Driven Expressive Talking Lips with Conditional Sequential Generative Adversarial Networkscited by 21
  12. 2020Re-synchronization using the Hand Preceding Model for Multi-modal Fusion in Automatic Continuous Cued Speech Recognitioncited by 17
  13. 2020Identity-Preserving Realistic Talking Face Generationcited by 12
  14. 2022Residual-guided Personalized Speech Synthesis based on Face Imagecited by 10
  15. 2021Cross-Modal Knowledge Distillation Method for Automatic Cued Speech Recognitioncited by 8
  16. 2023Memory-augmented Contrastive Learning for Talking Head Generationcited by 5

Built on by

Cited by year

Citing papers per year: 2023: 2; 2024: 3; 2025: 2.

7 papers.

  1. 2024FlowVQTalker: High-Quality Emotional Talking Face Generation through Normalizing Flow and Quantizationcited by 39
  2. 2024ScanTalk: 3D Talking Heads from Unregistered Scanscited by 24
  3. 2024Beyond Fixed Topologies: Unregistered Training and Comprehensive Evaluation Metrics for 3D Talking Headscited by 5
  4. 2023A Survey on Deep Multi-modal Learning for Body Language Recognition and Generationcited by 3
  5. 2023Realistic Speech-to-Face Generation with Speech-Conditioned Latent Diffusion Model with Face Priorcited by 1
  6. 2025Talking Head Generation via AU-Guided Landmark Predictioncited by 1
  7. 2025RealTalk: Realistic Emotion-Aware Lifelike Talking-Head Synthesiscited by 0