mlsanityData to 7 Oct 2026

Coherent3D: Coherent 3D Portrait Video Reconstruction via Triplane Fusion

arXiv 2412.08684 ↗First posted 11 Dec 2024cs.CV eess.IVcited by 0

Not on the map. Why?

Abstract (arXiv)

Recent breakthroughs in single-image 3D portrait reconstruction have enabled telepresence systems to stream 3D portrait videos from a single camera in real-time, democratizing telepresence. However, per-frame 3D reconstruction exhibits temporal inconsistency and forgets the user's appearance. On the other hand, self-reenactment methods can render coherent 3D portraits by driving a 3D avatar built from a single reference image, but fail to faithfully preserve the user's per-frame appearance (e.g., instantaneous facial expression and lighting). As a result, none of these two frameworks is an ideal solution for democratized 3D telepresence. In this work, we address this dilemma and propose a novel solution that maintains both coherent identity and dynamic per-frame appearance to enable the best possible realism. To this end, we propose a new fusion-based method that takes the best of both worlds by fusing a canonical 3D prior from a reference view with dynamic appearance from per-frame input views, producing temporally stable 3D videos with faithful reconstruction of the user's per-frame appearance. Trained only using synthetic data produced by an expression-conditioned 3D GAN, our encoder-based method achieves both state-of-the-art 3D reconstruction and temporal consistency on in-studio and in-the-wild datasets. https://research.nvidia.com/labs/amri/projects/coherent3d

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 58 of its 93 references that are in the map. First posted on arXiv 11 Dec 2024.

Builds on

58 of 93 references.

References · 58

  1. 20233D Gaussian Splatting for Real-Time Radiance Field Renderingcited by 6,141
  2. 2018A Style-Based Generator Architecture for Generative Adversarial Networkscited by 5,986
  3. 2020D-NeRF: Neural Radiance Fields for Dynamic Scenescited by 1,459
  4. 2021Efficient Geometry-aware 3D Generative Adversarial Networkscited by 1,204
  5. 2020Nerfies: Deformable Neural Radiance Fieldscited by 1,194
  6. 2020First Order Motion Model for Image Animationcited by 698
  7. 2021StyleNeRF: A Style-based 3D-Aware Generator for High-resolution Image Synthesiscited by 477
  8. 2020Learning an Animatable Detailed 3D Face Model from In-The-Wild Imagescited by 475
  9. 2020Dynamic Neural Radiance Fields for Monocular 4D Facial Avatar Reconstructioncited by 465
  10. 2020One-Shot Free-View Neural Talking-Head Synthesis for Video Conferencingcited by 438
  11. 2016Optical Flow Estimation using a Spatial Pyramid Networkcited by 421
  12. 2021StyleSDF: High-Resolution 3D-Consistent Image and Geometry Generationcited by 289
38 more
  1. 2019Few-Shot Adversarial Learning of Realistic Neural Talking Head Modelscited by 284
  2. 2020State of the Art on Neural Renderingcited by 276
  3. 2020StyleRig: Rigging StyleGAN for 3D Control over Portrait Imagescited by 267
  4. 2023GaussianAvatars: Photorealistic Head Avatars with Rigged 3D Gaussianscited by 256
  5. 2021I M Avatar: Implicit Morphable Head Avatars from Videoscited by 230
  6. 2021HeadNeRF: A Real-time NeRF-based Parametric Head Modelcited by 228
  7. 2020Disentangled and Controllable Face Image Generation via 3D Imitative-Contrastive Learningcited by 219
  8. 2021GRAM: Generative Radiance Manifolds for 3D-Aware Image Generationcited by 216
  9. 2022Thin-Plate Spline Motion Model for Image Animationcited by 209
  10. 2023NeRSemble: Multi-view Radiance Field Reconstruction of Human Headscited by 190
  11. 2022Depth-Aware Generative Adversarial Network for Talking Head Video Generationcited by 177
  12. 2022Latent Image Animator: Learning to Animate Images via Latent Space Navigationcited by 172
  13. 2022StyleHEAT: One-Shot High-Resolution Editable Talking Face Generation via Pre-trained StyleGANcited by 169
  14. 2022Recurrent Video Restoration Transformer with Guided Deformable Attentioncited by 168
  15. 2021CIPS-3D: A 3D-Aware Generator of GANs Based on Conditionally-Independent Pixel Synthesiscited by 149
  16. 2022MegaPortraits: One-shot Megapixel Neural Head Avatarscited by 131
  17. 2022RigNeRF: Fully Controllable Neural 3D Portraitscited by 126
  18. 2023Relightable Gaussian Codec Avatarscited by 126
  19. 2021Pixel Codec Avatarscited by 125
  20. 2022Next3D: Generative Neural Texture Rasterization for 3D-Aware Head Avatarscited by 124
  21. 2022Realistic One-shot Mesh-based Head Avatarscited by 119
  22. 2020Fast Bi-layer Neural Synthesis of One-Shot Realistic Head Avatarscited by 116
  23. 20213D-aware Image Synthesis via Learning Structural and Textural Representationscited by 109
  24. 2022EpiGRAF: Rethinking training of 3D GANscited by 108
  25. 2020HeadGAN: One-shot Neural Head Synthesis and Editingcited by 105
  26. 2024Real3D-Portrait: One-shot Realistic 3D Talking Portrait Synthesiscited by 97
  27. 2021MoFaNeRF: Morphable Facial Neural Radiance Fieldcited by 94
  28. 2021LOLNeRF: Learn from One Lookcited by 84
  29. 2024GPAvatar: Generalizable and Precise Head Avatar from Image(s)cited by 74
  30. 2022GRAM-HD: 3D-Consistent Image Generation at High Resolution with Generative Radiance Manifoldscited by 73
  31. 2022MetaPortrait: Identity-Preserving Talking Head Generation with Fast Personalized Adaptationcited by 69
  32. 2023One-Shot High-Fidelity Talking-Head Synthesis with Deformable Neural Radiance Fieldcited by 68
  33. 2023OTAvatar: One-shot Talking Face Avatar with Controllable Tri-plane Renderingcited by 67
  34. 2023Real-Time Radiance Fields for Single-Image Portrait View Synthesiscited by 64
  35. 20223D GAN Inversion for Controllable Portrait Image Animationcited by 53
  36. 2023NOFA: NeRF-based One-shot Facial Avatar Reconstructioncited by 53
  37. 2023StyleAvatar: Real-time Photo-realistic Portrait Avatar from a Single Videocited by 52
  38. 20223D GAN Inversion with Pose Optimizationcited by 44

50 most cited of 58 All 58 (JSON)

Built on by

Not cited yet.