mlsanityData to 7 Oct 2026

PIRenderer: Controllable Portrait Image Generation via Semantic Neural Rendering

arXiv 2109.08379 ↗First posted 17 Sep 2021cs.CV cs.AIcited by 202

Not on the map. Why?

Cited by 31 in Diffusion models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Generating portrait images by controlling the motions of existing faces is an important task of great consequence to social media industries. For easy use and intuitive control, semantically meaningful and fully disentangled parameters should be used as modifications. However, many existing techniques do not provide such fine-grained controls or use indirect editing methods i.e. mimic motions of other individuals. In this paper, a Portrait Image Neural Renderer (PIRenderer) is proposed to control the face motions with the parameters of three-dimensional morphable face models (3DMMs). The proposed model can generate photo-realistic portrait images with accurate movements according to intuitive modifications. Experiments on both direct and indirect editing tasks demonstrate the superiority of this model. Meanwhile, we further extend this model to tackle the audio-driven facial reenactment task by extracting sequential motions from audio inputs. We show that our model can generate coherent videos with convincing movements from only a single reference image and a driving audio stream. Our source code is available at https://github.com/RenYurui/PIRender.

Key facts. Cited by 202 papers in the map (data through 7 Oct 2026). Builds on 40 of its 54 references that are in the map. First posted on arXiv 17 Sep 2021.

Builds on

40 of 54 references.

References · 40

  1. 2018The Unreasonable Effectiveness of Deep Features as a Perceptual Metriccited by 11,029
  2. 2018A Style-Based Generator Architecture for Generative Adversarial Networkscited by 5,986
  3. 2016Image-to-Image Translation with Conditional Adversarial Networkscited by 4,510
  4. 2016Perceptual Losses for Real-Time Style Transfer and Super-Resolutioncited by 2,678
  5. 2018Large Scale GAN Training for High Fidelity Natural Image Synthesiscited by 2,609
  6. 2020RAFT: Recurrent All-Pairs Field Transforms for Optical Flowcited by 2,385
  7. 2015Variational Inference with Normalizing Flowscited by 2,161
  8. 2017Arbitrary Style Transfer in Real-time with Adaptive Instance Normalizationcited by 1,998
  9. 2018Glow: Generative Flow with Invertible 1x1 Convolutionscited by 1,539
  10. 2014NICE: Non-linear Independent Components Estimationcited by 1,052
  11. 2017StarGAN: Unified Generative Adversarial Networks for Multi-Domain Image-to-Image Translationcited by 1,036
  12. 2017VoxCeleb: a large-scale speaker identification datasetcited by 714
28 more
  1. 2020First Order Motion Model for Image Animationcited by 698
  2. 2019Image2StyleGAN: How to Embed Images Into the StyleGAN Latent Space?cited by 675
  3. 2020Face2Face: Real-time Face Capture and Reenactment of RGB Videoscited by 631
  4. 2019Accurate 3D Face Reconstruction with Weakly-Supervised Learning: From Single Image to Image Setcited by 481
  5. 2020One-Shot Free-View Neural Talking-Head Synthesis for Video Conferencingcited by 438
  6. 2018Video-to-Video Synthesiscited by 388
  7. 2019Few-Shot Adversarial Learning of Realistic Neural Talking Head Modelscited by 284
  8. 2020StyleRig: Rigging StyleGAN for 3D Control over Portrait Imagescited by 267
  9. 2018Deep Video Portraitscited by 265
  10. 2019Deferred Neural Rendering: Image Synthesis using Neural Texturescited by 253
  11. 2019Hierarchical Cross-Modal Talking Face Generationwith Dynamic Pixel-Wise Losscited by 252
  12. 2019Neural Voice Puppetry: Audio-driven Facial Reenactmentcited by 249
  13. 2020Disentangled and Controllable Face Image Generation via 3D Imitative-Contrastive Learningcited by 219
  14. 2018Talking Face Generation by Adversarially Disentangled Audio-Visual Representationcited by 212
  15. 2018Animating Arbitrary Objects via Deep Motion Transfercited by 206
  16. 2019Few-shot Video-to-Video Synthesiscited by 196
  17. 2018X2Face: A network for controlling face generation by using images, audio, and pose codescited by 189
  18. 2020What Matters in Unsupervised Optical Flowcited by 124
  19. 2020Fast Bi-layer Neural Synthesis of One-Shot Realistic Head Avatarscited by 116
  20. 2020Neural Head Reenactment with Latent Pose Descriptorscited by 106
  21. 2020Deep Image Spatial Transformation for Person Image Generationcited by 100
  22. 2017You said that?cited by 99
  23. 2018Recycle-GAN: Unsupervised Video Retargetingcited by 89
  24. 2018ReenactGAN: Learning to Reenact Faces via Boundary Transfercited by 85
  25. 2018Talking Face Generation by Conditional Recurrent Adversarial Networkcited by 84
  26. 2019MoGlow: Probabilistic and controllable motion synthesis using normalising flowscited by 47
  27. 2020Deep Spatial Transformation for Pose-Guided Person Image Generation and Animationcited by 43
  28. 2018End-to-End Speech-Driven Facial Animation with Temporal GANscited by 31

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 2; 2022: 30; 2023: 54; 2024: 62; 2025: 32; 2026: 22.

202 papers.

Sort:

By area:

  1. 2022SadTalker: Learning Realistic 3D Motion Coefficients for Stylized Audio-Driven Single Image Talking Face Animationcited by 380
  2. 2023MagicAnimate: Temporally Consistent Human Image Animation using Diffusion Modelcited by 330
  3. 2024Hallo: Hierarchical Audio-Driven Visual Synthesis for Portrait Image Animationcited by 198
  4. 2024VASA-1: Lifelike Audio-Driven Talking Faces Generated in Real Timecited by 186
  5. 2022StyleHEAT: One-Shot High-Resolution Editable Talking Face Generation via Pre-trained StyleGANcited by 169
  6. 2023Diffused Heads: Diffusion Models Beat GANs on Talking-Face Generationcited by 142
  7. 2024DF40: Toward Next-Generation Deepfake Detectioncited by 126
  8. 2022Next3D: Generative Neural Texture Rasterization for 3D-Aware Head Avatarscited by 124
  9. 2024Hallo3: Highly Dynamic and Realistic Portrait Image Animation with Video Diffusion Transformercited by 121
  10. 2022VideoReTalking: Audio-based Lip Synchronization for Talking Head Video Editing In the Wildcited by 106
  11. 2024Loopy: Taming Audio-Driven Portrait Avatar with Long-Term Motion Dependencycited by 102
  12. 2022Generative Neural Articulated Radiance Fieldscited by 101
38 more
  1. 2023Identity-Preserving Talking Face Generation with Landmark and Appearance Priorscited by 97
  2. 2024Generalizable and Animatable Gaussian Head Avatarcited by 95
  3. 2022Progressive Disentangled Representation Learning for Fine-Grained Controllable Talking Head Synthesiscited by 93
  4. 2023StyleTalk: One-shot Talking Head Generation with Controllable Speaking Stylescited by 92
  5. 2024EDTalk: Efficient Disentanglement for Emotional Talking Head Synthesiscited by 79
  6. 2024GPAvatar: Generalizable and Precise Head Avatar from Image(s)cited by 74
  7. 2023DINet: Deformation Inpainting Network for Realistic Face Visually Dubbing on High Resolution Videocited by 73
  8. 2022MetaPortrait: Identity-Preserving Talking Head Generation with Fast Personalized Adaptationcited by 69
  9. 2023One-Shot High-Fidelity Talking-Head Synthesis with Deformable Neural Radiance Fieldcited by 68
  10. 2021Responsive Listening Head Generation: A Benchmark Dataset and Baselinecited by 67
  11. 2023OTAvatar: One-shot Talking Face Avatar with Controllable Tri-plane Renderingcited by 67
  12. 2024Face Adapter for Pre-Trained Diffusion Models with Fine-Grained ID and Attribute Controlcited by 66
  13. 2024Portrait4D-v2: Pseudo Multi-View Data Creates Better 4D Head Synthesizercited by 64
  14. 2023Portrait4D: Learning One-Shot 4D Head Avatar Synthesis using Synthetic Datacited by 55
  15. 20223D GAN Inversion for Controllable Portrait Image Animationcited by 53
  16. 2023NOFA: NeRF-based One-shot Facial Avatar Reconstructioncited by 53
  17. 2023StyleAvatar: Real-time Photo-realistic Portrait Avatar from a Single Videocited by 52
  18. 2024CAP4D: Creating Animatable 4D Portrait Avatars with Morphable Multi-View Diffusion Modelscited by 52
  19. 2023LatentAvatar: Learning Latent Expression Code for Expressive Neural Head Avatarcited by 50
  20. 2023Face Animation with an Attribute-Guided Diffusion Modelcited by 48
  21. 2023HyperReenact: One-Shot Reenactment via Jointly Learning to Refine and Retarget Facescited by 48
  22. 2024VLOGGER: Multimodal Diffusion for Embodied Avatar Synthesiscited by 48
  23. 2023VividTalk: One-Shot Audio-Driven Talking Head Generation Based on 3D Hybrid Priorcited by 47
  24. 2022AniFaceGAN: Animatable 3D-Aware Face Image Generation for Video Avatarscited by 46
  25. 2023DPE: Disentanglement of Pose and Expression for General Video Portrait Editingcited by 45
  26. 2024HeadGAP: Few-Shot 3D Head Avatar via Generalizable Gaussian Priorscited by 44
  27. 2023GAIA: Zero-shot Talking Avatar Generationcited by 43
  28. 2025LAM: Large Avatar Model for One-shot Animatable Gaussian Headcited by 43
  29. 2023High-Fidelity and Freely Controllable Talking Head Video Generationcited by 42
  30. 2024INFP: Audio-Driven Interactive Head Generation in Dyadic Conversationscited by 40
  31. 2024FlowVQTalker: High-Quality Emotional Talking Face Generation through Normalizing Flow and Quantizationcited by 39
  32. 2022FDNeRF: Few-shot Dynamic Neural Radiance Fields for Face Reconstruction and Expression Editingcited by 38
  33. 2022Controllable 3D Face Synthesis with Conditional Generative Occupancy Fieldscited by 37
  34. 2023High-fidelity Generalized Emotional Talking Face Generation with Multi-modal Emotion Space Learningcited by 37
  35. 2025DreamActor-M1: Holistic, Expressive and Robust Human Image Animation with Hybrid Guidancecited by 36
  36. 2023VOODOO 3D: Volumetric Portrait Disentanglement for One-Shot 3D Head Reenactmentcited by 33
  37. 2023TalkCLIP: Talking Head Generation with Text-Guided Expressive Speaking Stylescited by 32
  38. 2024Dyadic Interaction Modeling for Social Behavior Generationcited by 32

50 most cited of 202 All 202 (JSON)