mlsanityData to 7 Oct 2026

RodinHD: High-Fidelity 3D Avatar Generation with Diffusion Models

arXiv 2407.06938 ↗First posted 9 Jul 2024cs.CVcited by 36

On the map: Diffusion models → 3D, point cloud and scene generation

Abstract (arXiv)

We present RodinHD, which can generate high-fidelity 3D avatars from a portrait image. Existing methods fail to capture intricate details such as hairstyles which we tackle in this paper. We first identify an overlooked problem of catastrophic forgetting that arises when fitting triplanes sequentially on many avatars, caused by the MLP decoder sharing scheme. To overcome this issue, we raise a novel data scheduling strategy and a weight consolidation regularization term, which improves the decoder's capability of rendering sharper details. Additionally, we optimize the guiding effect of the portrait image by computing a finer-grained hierarchical representation that captures rich 2D texture cues, and injecting them to the 3D diffusion model at multiple layers via cross-attention. When trained on 46K avatars with a noise schedule optimized for triplanes, the resulting model can generate 3D avatars with notably better details than previous methods and can generalize to in-the-wild portrait input.

Key facts. Cited by 36 papers in the map (data through 7 Oct 2026). Placed in 3D, point cloud and scene generation, one of 14 directions in Diffusion models. Builds on 73 of its 84 references that are in the map. First posted on arXiv 9 Jul 2024.

Builds on

73 of 84 references.

Closest ancestors · 11

Same direction, newest first

  1. 2023DreamCraft3D: Hierarchical 3D Generation with Bootstrapped Diffusion Prior25 Oct
  2. 2023AutoDecoding Latent 3D Diffusion Models7 Jul
  3. 2023Single-Stage Diffusion NeRF: A Unified Approach to 3D Generation and Reconstruction13 Apr
  4. 2023Make-It-3D: High-Fidelity 3D Creation from A Single Image with Diffusion Prior24 Mar
  5. 20233DGen: Triplane Latent Diffusion for Textured Mesh Generation9 Mar
  6. 2022Rodin: A Generative Model for Sculpting 3D Digital Avatars Using Diffusion12 Dec
  7. 2022DiffRF: Rendering-Guided 3D Radiance Field Diffusion2 Dec
  8. 20223D Neural Field Generation using Triplane Diffusion30 Nov
  9. 2022Magic3D: High-Resolution Text-to-3D Content Creation18 Nov
  10. 2022LION: Latent Point Diffusion Models for 3D Shape Generation12 Oct
  11. 2022DreamFusion: Text-to-3D using 2D Diffusion29 Sep

Other references · 62

  1. 2017Attention Is All You Needcited by 59,932
  2. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  3. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  4. 2020Denoising Diffusion Probabilistic Modelscited by 20,851
  5. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  6. 2014Very Deep Convolutional Networks for Large-Scale Image Recognitioncited by 15,267
  7. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  8. 2018The Unreasonable Effectiveness of Deep Features as a Perceptual Metriccited by 11,029
  9. 2020Score-Based Generative Modeling through Stochastic Differential Equationscited by 8,964
  10. 2021Diffusion Models Beat GANs on Image Synthesiscited by 8,185
  11. 2022Photorealistic Text-to-Image Diffusion Models with Deep Language Understandingcited by 6,322
  12. 2018A Style-Based Generator Architecture for Generative Adversarial Networkscited by 5,986
38 more
  1. 2022Generative Adversarial Networkscited by 5,603
  2. 2023Adding Conditional Control to Text-to-Image Diffusion Modelscited by 5,466
  3. 2016Overcoming catastrophic forgetting in neural networkscited by 4,718
  4. 2023SDXL: Improving Latent Diffusion Models for High-Resolution Image Synthesiscited by 4,290
  5. 2021Improved Denoising Diffusion Probabilistic Modelscited by 3,736
  6. 2022Instant Neural Graphics Primitives with a Multiresolution Hash Encodingcited by 3,698
  7. 2021GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Modelscited by 3,595
  8. 2019DeepSDF: Learning Continuous Signed Distance Functions for Shape Representationcited by 2,682
  9. 2020Implicit Neural Representations with Periodic Activation Functionscited by 2,214
  10. 2021Mip-NeRF 360: Unbounded Anti-Aliased Neural Radiance Fieldscited by 2,013
  11. 2021NeuS: Learning Neural Implicit Surfaces by Volume Rendering for Multi-view Reconstructioncited by 1,701
  12. 2021Efficient Geometry-aware 3D Generative Adversarial Networkscited by 1,204
  13. 2021Cascaded Diffusion Models for High Fidelity Image Generationcited by 1,155
  14. 2019An Introduction to Variational Autoencoderscited by 997
  15. 2020GIRAFFE: Representing Scenes as Compositional Generative Neural Feature Fieldscited by 784
  16. 2021Vector Quantized Diffusion Model for Text-to-Image Synthesiscited by 773
  17. 2020pi-GAN: Periodic Implicit Generative Adversarial Networks for 3D-Aware Image Synthesiscited by 708
  18. 2020GRAF: Generative Radiance Fields for 3D-Aware Image Synthesiscited by 688
  19. 2016Learning a Probabilistic Latent Space of Object Shapes via 3D Generative-Adversarial Modelingcited by 558
  20. 2019Accurate 3D Face Reconstruction with Weakly-Supervised Learning: From Single Image to Image Setcited by 481
  21. 2022GET3D: A Generative Model of High Quality 3D Textured Shapes Learned from Imagescited by 446
  22. 2023Simple diffusion: End-to-end diffusion for high resolution imagescited by 413
  23. 2023Shap-E: Generating Conditional 3D Implicit Functionscited by 379
  24. 2021Towards Real-World Blind Face Restoration with Generative Facial Priorcited by 347
  25. 2023Common Diffusion Noise Schedules and Sample Steps are Flawedcited by 308
  26. 2023Efficient Diffusion Training via Min-SNR Weighting Strategycited by 242
  27. 2021GRAM: Generative Radiance Manifolds for 3D-Aware Image Generationcited by 216
  28. 2021Fake It Till You Make It: Face analysis in the wild using synthetic data alonecited by 200
  29. 2023On the Importance of Noise Scheduling for Diffusion Modelscited by 193
  30. 2022Scalable Adaptive Computation for Iterative Generationcited by 190
  31. 2024MotionLCM: Real-time Controllable Motion Generation via Latent Consistency Modelcited by 164
  32. 2021StyleSwin: Transformer-based GAN for High-resolution Image Generationcited by 152
  33. 2022GAUDI: A Neural Architect for Immersive 3D Scene Generationcited by 136
  34. 2016Generative and Discriminative Voxel Modeling with Convolutional Neural Networkscited by 128
  35. 2019Towards Unsupervised Learning of Generative Models for 3D Controllable Image Synthesiscited by 104
  36. 2023Generative Image Dynamicscited by 93
  37. 2023Relay Diffusion: Unifying diffusion process across resolutions for image synthesiscited by 78
  38. 2022GRAM-HD: 3D-Consistent Image Generation at High Resolution with Generative Radiance Manifoldscited by 73

50 most cited of 62 All 62 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 7; 2025: 18; 2026: 11.

36 papers.

Sort:

By area:

  1. 2024Structured 3D Latents for Scalable and Versatile 3D Generationcited by 742
  2. 2024CAP4D: Creating Animatable 4D Portrait Avatars with Morphable Multi-View Diffusion Modelscited by 52
  3. 2025DiffSplat: Repurposing Image Diffusion Models for Scalable Gaussian Splat Generationcited by 49
  4. 2024Efficient Diffusion Models: A Comprehensive Survey from Principles to Practicescited by 39
  5. 2025Gaussian Variation Field Diffusion for High-fidelity Video-to-4D Synthesiscited by 33
  6. 2024FaceLift: Learning Generalizable Single Image 3D Face Reconstruction from Synthetic Headscited by 22
  7. 2024Joker: Conditional 3D Head Synthesis with Extreme Facial Expressionscited by 17
  8. 2025Arc2Avatar: Generating Expressive 3D Avatars from a Single Image via ID Guidancecited by 16
  9. 2025AvatarArtist: Open-Domain 4D Avatarizationcited by 15
  10. 2025Synthetic Prior for Few-Shot Drivable Head Avatar Inversioncited by 14
  11. 2025TripNet: Learning Large-scale High-fidelity 3D Car Aerodynamics with Triplane Networkscited by 14
  12. 2025SOAP: Style-Omniscient Animatable Portraitscited by 11
24 more
  1. 2025MoGA: 3D Generative Avatar Prior for Monocular Gaussian Avatar Reconstructioncited by 8
  2. 2025PanoLAM: Large Avatar Model for Gaussian Full-Head Synthesis from One-shot Unposed Imagecited by 7
  3. 2025PercHead: Perceptual Head Model for Single-Image 3D Head Reconstruction & Editingcited by 6
  4. 2024MixedGaussianAvatar: Realistically and Geometrically Accurate Head Avatar via Mixed 2D-3D Gaussianscited by 4
  5. 2025FRIDU: Functional Map Refinement with Guided Image Diffusioncited by 4
  6. 2025Controlling Avatar Diffusion with Learnable Gaussian Embeddingcited by 3
  7. 2025SpinMeRound: Consistent Multi-View Identity Generation Using Diffusion Modelscited by 3
  8. 2025Bringing Your Portrait to 3D Presencecited by 3
  9. 2026Sculpt4D: Generating 4D Shapes via Sparse-Attention Diffusion Transformerscited by 3
  10. 2026Mix3R: Mixing Feed-forward Reconstruction and Generative 3D Priors for Joint Multi-view Aligned 3D Reconstruction and Pose Estimationcited by 3
  11. 2025SVAD: From Single Image to 3D Avatar via Synthetic Data Generation with Video Diffusion and Data Augmentationcited by 2
  12. 2025ToonifyGB: StyleGAN-based Gaussian Blendshapes for 3D Stylized Head Avatarscited by 2
  13. 2025T2Bs: Text-to-Character Blendshapes via Video Generationcited by 2
  14. 2024Towards High-Fidelity 3D Portrait Generation with Rich Details by Cross-View Prior-Aware Diffusioncited by 1
  15. 2026Any3DAvatar: Fast and High-Quality Full-Head 3D Avatar Reconstruction from Single Portrait Imagecited by 1
  16. 2026GASE: Gaussian Splatting-Based Automated System for Reconstructing Embodied-Simulation Environmentscited by 1
  17. 2025Dream3DAvatar: Text-Controlled 3D Avatar Reconstruction from a Single Imagecited by 0
  18. 2026OMEGA-Avatar: One-shot Modeling of 360deg Gaussian Avatarscited by 0
  19. 2026GenLCA: 3D Diffusion for Full-Body Avatars from In-the-Wild Videoscited by 0
  20. 2026Physically Grounded 3D Generative Reconstruction under Hand Occlusion using Proprioception and Multi-Contact Touchcited by 0
  21. 20263DRealHead: Few-Shot Detailed Head Avatarcited by 0
  22. 2026Large-Scale High-Quality 3D Gaussian Head Reconstruction from Multi-View Capturescited by 0
  23. 2026Photorealistic Novel View Synthesis of Human Faces using Next-Scale Transformerscited by 0
  24. 2026Source-Face Authenticity Detection for 3D Gaussian Heads Reconstructed from a Single Portrait: A Benchmark and Dedicated Detectorcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Instant 3D Human Avatar Generation using Image Diffusion Models
  2. 2025TurboPortrait3D: Single-step diffusion-based fast portrait novel-view synthesis
  3. 2023StyleAvatar3D: Leveraging Image-Text Diffusion Models for High-Fidelity 3D Avatar Generation
  4. 2024Human-3Diffusion: Realistic Avatar Creation via Explicit 3D Consistent Diffusion Models
  5. 2024Gen-3Diffusion: Realistic Image-to-3D Generation via 2D & 3D Diffusion Synergy
  6. 2024Morphable Diffusion: 3D-Consistent Diffusion for Single-image Avatar Creation
  7. 2024AvatarMMC: 3D Head Avatar Generation and Editing with Multi-Modal Conditioning
  8. 2023HeadSculpt: Crafting 3D Head Avatars with Text