mlsanityData to 7 Oct 2026

Audio-Driven Talking Face Video Generation with Dynamic Convolution Kernels

arXiv 2201.05986 ↗First posted 16 Jan 2022cs.CV cs.MMcited by 23

Not on the map. Why?

Abstract (arXiv)

In this paper, we present a dynamic convolution kernel (DCK) strategy for convolutional neural networks. Using a fully convolutional network with the proposed DCKs, high-quality talking-face video can be generated from multi-modal sources (i.e., unmatched audio and video) in real time, and our trained model is robust to different identities, head postures, and input audios. Our proposed DCKs are specially designed for audio-driven talking face video generation, leading to a simple yet effective end-to-end system. We also provide a theoretical analysis to interpret why DCKs work. Experimental results show that our method can generate high-quality talking-face video with background at 60 fps. Comparison and evaluation between our method and the state-of-the-art methods demonstrate the superiority of our method.

Key facts. Cited by 23 papers in the map (data through 7 Oct 2026). Builds on 28 of its 52 references that are in the map. First posted on arXiv 16 Jan 2022.

Builds on

28 of 52 references.

References · 28

  1. 2015U-Net: Convolutional Networks for Biomedical Image Segmentationcited by 19,604
  2. 2014Very Deep Convolutional Networks for Large-Scale Image Recognitioncited by 15,267
  3. 2018The Unreasonable Effectiveness of Deep Features as a Perceptual Metriccited by 11,029
  4. 2017Multimodal Machine Learning: A Survey and Taxonomycited by 1,004
  5. 2016HyperNetworkscited by 1,003
  6. 2016Least Squares Generative Adversarial Networkscited by 1,000
  7. 2020A Lip Sync Expert Is All You Need for Speech to Lip Generation In The Wildcited by 669
  8. 2020Face2Face: Real-time Face Capture and Reenactment of RGB Videoscited by 631
  9. 2019Dynamic Convolution: Attention over Convolution Kernelscited by 355
  10. 2019Few-Shot Adversarial Learning of Realistic Neural Talking Head Modelscited by 284
  11. 2018Deep Video Portraitscited by 265
  12. 2019Hierarchical Cross-Modal Talking Face Generationwith Dynamic Pixel-Wise Losscited by 252
16 more
  1. 2019Neural Voice Puppetry: Audio-driven Facial Reenactmentcited by 249
  2. 2018Talking Face Generation by Adversarially Disentangled Audio-Visual Representationcited by 212
  3. 2018X2Face: A network for controlling face generation by using images, audio, and pose codescited by 189
  4. 2018GANimation: Anatomically-aware Facial Animation from a Single Imagecited by 160
  5. 2018Lip Movements Generation at a Glancecited by 154
  6. 2019Realistic Speech-Driven Facial Animation with GANscited by 142
  7. 2019Text-based Editing of Talking-head Videocited by 102
  8. 2017You said that?cited by 99
  9. 2018Talking Face Generation by Conditional Recurrent Adversarial Networkcited by 84
  10. 2020Everybody's Talkin': Let Me Talk as You Wantcited by 83
  11. 2017Attention-Based Multimodal Fusion for Video Descriptioncited by 75
  12. 2018End-to-End Speech-Driven Facial Animation with Temporal GANscited by 31
  13. 2019Dynamic Kernel Distillation for Efficient Pose Estimation in Videoscited by 30
  14. 2019One-shot Face Reenactmentcited by 21
  15. 2017Modeling Multimodal Clues in a Hybrid Deep Learning Framework for Video Classificationcited by 13
  16. 2020Deep Multimodality Learning for UAV Video Aesthetic Quality Assessmentcited by 9

Built on by

Cited by year

Citing papers per year: 2022: 5; 2023: 9; 2024: 7; 2025: 2.

23 papers.

Sort:

By area:

  1. 2024Real3D-Portrait: One-shot Realistic 3D Talking Portrait Synthesiscited by 97
  2. 2023TalkCLIP: Talking Head Generation with Text-Guided Expressive Speaking Stylescited by 32
  3. 2024Say Anything with Any Stylecited by 26
  4. 2022Deep Learning for Visual Speech Analysis: A Surveycited by 21
  5. 2024MimicTalk: Mimicking a personalized and expressive 3D talking face in minutescited by 16
  6. 2022Language-Guided Face Animation by Recurrent StyleGAN-based Generatorcited by 9
  7. 2022StableFace: Analyzing and Improving Motion Stability for Talking Face Generationcited by 9
  8. 2023SingingHead: A Large-scale 4D Dataset for Singing Head Animationcited by 9
  9. 2023MMFace4D: A Large-Scale Multi-Modal 4D Face Dataset for Audio-Driven 3D Face Animationcited by 8
  10. 2023Emotional Talking Head Generation based on Memory-Sharing and Attention-Augmented Networkscited by 7
  11. 2023Parametric Implicit Face Representation for Audio-Driven Facial Reenactmentcited by 7
  12. 2023Temporally-Adaptive Models for Efficient Video Understandingcited by 6
11 more
  1. 2023HyperLips: Hyper Control Lips with High Resolution Decoder for Talking Face Generationcited by 6
  2. 2023PMMTalk: Speech-Driven 3D Facial Animation from Complementary Pseudo Multi-modal Featurescited by 6
  3. 2024Style-Preserving Lip Sync via Audio-Aware Style Referencecited by 5
  4. 2024PortraitTalk: Towards Customizable One-Shot Audio-to-Talking Face Generationcited by 5
  5. 2022Dynamic Neural Textures: Generating Talking-Face Videos with Continuously Controllable Expressionscited by 4
  6. 2024EmoSpeaker: One-shot Fine-grained Emotion-Controlled Talking Face Generationcited by 4
  7. 2024Rethinking Human Evaluation Protocol for Text-to-Video Models: Enhancing Reliability,Reproducibility, and Practicalitycited by 4
  8. 2025Exploiting Temporal Audio-Visual Correlation Embedding for Audio-Driven One-Shot Talking Head Animationcited by 3
  9. 2022Fighting Malicious Media Data: A Survey on Tampering Detection and Deepfake Detectioncited by 2
  10. 2023Fast Fourier Inception Networks for Occluded Video Predictioncited by 1
  11. 2025MoDA: Multi-modal Diffusion Architecture for Talking Head Generationcited by 0