mlsanityData to 7 Oct 2026

Audio-Visual Face Reenactment

arXiv 2210.02755 ↗First posted 6 Oct 2022cs.CVcited by 18

Not on the map. Why?

Abstract (arXiv)

This work proposes a novel method to generate realistic talking head videos using audio and visual streams. We animate a source image by transferring head motion from a driving video using a dense motion field generated using learnable keypoints. We improve the quality of lip sync using audio as an additional input, helping the network to attend to the mouth region. We use additional priors using face segmentation and face mesh to improve the structure of the reconstructed faces. Finally, we improve the visual quality of the generations by incorporating a carefully designed identity-aware generator module. The identity-aware generator takes the source image and the warped motion features as input to generate a high-quality output with fine-grained details. Our method produces state-of-the-art results and generalizes well to unseen faces, languages, and voices. We comprehensively evaluate our approach using multiple metrics and outperforming the current techniques both qualitative and quantitatively. Our work opens up several applications, including enabling low bandwidth video calls. We release a demo video and additional information at http://cvit.iiit.ac.in/research/projects/cvit-projects/avfr.

Key facts. Cited by 18 papers in the map (data through 7 Oct 2026). Builds on 33 of its 43 references that are in the map. First posted on arXiv 6 Oct 2022.

Builds on

33 of 43 references.

References · 33

  1. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  2. 2015U-Net: Convolutional Networks for Biomedical Image Segmentationcited by 19,604
  3. 2018The Unreasonable Effectiveness of Deep Features as a Perceptual Metriccited by 11,029
  4. 2015Rethinking the Inception Architecture for Computer Visioncited by 5,451
  5. 2021Zero-Shot Text-to-Image Generationcited by 4,276
  6. 2018Spectral Normalization for Generative Adversarial Networkscited by 1,618
  7. 2019Semantic Image Synthesis with Spatially-Adaptive Normalizationcited by 1,295
  8. 2016Instance Normalization: The Missing Ingredient for Fast Stylizationcited by 1,187
  9. 2016Generative Adversarial Text to Image Synthesiscited by 735
  10. 2017VoxCeleb: a large-scale speaker identification datasetcited by 714
  11. 2020First Order Motion Model for Image Animationcited by 698
  12. 2020A Lip Sync Expert Is All You Need for Speech to Lip Generation In The Wildcited by 669
21 more
  1. 2019MediaPipe: A Framework for Building Perception Pipelinescited by 572
  2. 2017How far are we from solving the 2D & 3D Face Alignment problem? (and a dataset of 230,000 3D facial landmarks)cited by 524
  3. 2020One-Shot Free-View Neural Talking-Head Synthesis for Video Conferencingcited by 438
  4. 2020MakeItTalk: Speaker-Aware Talking-Head Animationcited by 310
  5. 2021Pose-Controllable Talking Face Generation by Implicitly Modularized Audio-Visual Representationcited by 288
  6. 2019Few-Shot Adversarial Learning of Realistic Neural Talking Head Modelscited by 284
  7. 2021Motion Representations for Articulated Animationcited by 253
  8. 2018Talking Face Generation by Adversarially Disentangled Audio-Visual Representationcited by 212
  9. 2021PIRenderer: Controllable Portrait Image Generation via Semantic Neural Renderingcited by 202
  10. 2021Audio-Driven Emotional Video Portraitscited by 180
  11. 2022Depth-Aware Generative Adversarial Network for Talking Head Video Generationcited by 177
  12. 2021Audio2Head: Audio-driven One-shot Talking-head Generation with Natural Head Motioncited by 148
  13. 2017You said that?cited by 99
  14. 2020Towards Automatic Face-to-Face Translationcited by 95
  15. 2018Talking Face Generation by Conditional Recurrent Adversarial Networkcited by 84
  16. 2019Unsupervised Part-Based Disentangling of Object Shape and Appearancecited by 67
  17. 2019ICface: Interpretable and Controllable Face Reenactment Using GANscited by 50
  18. 2021Write-a-speaker: Text-based Emotional and Rhythmic Talking-head Generationcited by 46
  19. 2021Live Speech Portraits: Real-Time Photorealistic Talking-Head Animationcited by 36
  20. 2020Faces `a la Carte: Text-to-Face Generation via Attribute Disentanglementcited by 16
  21. 2021Txt2Vid: Ultra-Low Bitrate Compression of Talking-Head Videos via Textcited by 9

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 1; 2023: 3; 2024: 8; 2025: 5; 2026: 1.

18 papers.

Sort:

By area:

  1. 2023AV-Deepfake1M: A Large-Scale LLM-Driven Audio-Visual Deepfake Datasetcited by 79
  2. 2024Deepfake Generation and Detection: A Benchmark and Surveycited by 66
  3. 2024Face Adapter for Pre-Trained Diffusion Models with Fine-Grained ID and Attribute Controlcited by 66
  4. 2024Audio-Visual Segmentation via Unlabeled Frame Exploitationcited by 27
  5. 2024FSRT: Facial Scene Representation Transformer for Face Reenactment from Factorized Appearance, Head-pose, and Facial Expression Featurescited by 14
  6. 2022Compressing Video Calls using Synthetic Talking Headscited by 10
  7. 2023From Pixels to Portraits: A Comprehensive Survey of Talking Head Generation Techniques and Applicationscited by 9
  8. 2025Advancing Talking Head Generation: A Comprehensive Survey of Multi-Modal Methodologies, Datasets, Evaluation Metrics, and Loss Functionscited by 6
  9. 2024A Hitchhikers Guide to Fine-Grained Face Forgery Detection Using Common Sense Reasoningcited by 4
  10. 2024Anchored Diffusion for Video Face Reenactmentcited by 2
  11. 2024MIMAFace: Face Animation via Motion-Identity Modulated Appearance Feature Learningcited by 2
  12. 2025Total-Editing: Head Avatar with Editable Appearance, Motion, and Lightingcited by 2
6 more
  1. 2025Unmasking Puppeteers: Leveraging Biometric Leakage to Expose Impersonation in AI-Based Videoconferencingcited by 2
  2. 2025GaussianHeadTalk: Wobble-Free 3D Talking Heads with Audio Driven Gaussian Splattingcited by 2
  3. 2023Learning and Evaluating Human Preferences for Conversational Head Generationcited by 1
  4. 2025Learning Semantic Facial Descriptors for Accurate Face Animationcited by 1
  5. 2024Maia: A Real-time Non-Verbal Chat for Human-AI Interactioncited by 0
  6. 2026LPIPS-AttnWav2Lip: Generic Audio-Driven lip synchronization for Talking Head Generation in the Wildcited by 0