mlsanityData to 7 Oct 2026

Score Jacobian Chaining: Lifting Pretrained 2D Diffusion Models for 3D Generation

arXiv 2212.00774 ↗First posted 1 Dec 2022cs.CV cs.LGcited by 597

On the map: Diffusion models → 3D, point cloud and scene generation

Abstract (arXiv)

A diffusion model learns to predict a vector field of gradients. We propose to apply chain rule on the learned gradients, and back-propagate the score of a diffusion model through the Jacobian of a differentiable renderer, which we instantiate to be a voxel radiance field. This setup aggregates 2D scores at multiple camera viewpoints into a 3D score, and repurposes a pretrained 2D model for 3D data generation. We identify a technical challenge of distribution mismatch that arises in this application, and propose a novel estimation mechanism to resolve it. We run our algorithm on several off-the-shelf diffusion image generative models, including the recently released Stable Diffusion trained on the large-scale LAION dataset.

Key facts. Cited by 597 papers in the map (data through 7 Oct 2026). Placed in 3D, point cloud and scene generation, one of 14 directions in Diffusion models. Builds on 53 of its 74 references that are in the map. First posted on arXiv 1 Dec 2022.

Builds on

53 of 74 references.

Closest ancestors · 1

Same direction, newest first

  1. 2022DreamFusion: Text-to-3D using 2D Diffusion29 Sep

Other references · 52

  1. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  3. 2020Denoising Diffusion Probabilistic Modelscited by 20,851
  4. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  5. 2020Denoising Diffusion Implicit Modelscited by 9,655
  6. 2020Score-Based Generative Modeling through Stochastic Differential Equationscited by 8,964
  7. 2021Diffusion Models Beat GANs on Image Synthesiscited by 8,185
  8. 2015Deep Unsupervised Learning using Nonequilibrium Thermodynamicscited by 7,005
  9. 2022Hierarchical Text-Conditional Image Generation with CLIP Latentscited by 6,539
  10. 2022Photorealistic Text-to-Image Diffusion Models with Deep Language Understandingcited by 6,322
  11. 2018A Style-Based Generator Architecture for Generative Adversarial Networkscited by 5,986
  12. 2022Classifier-Free Diffusion Guidancecited by 5,717
38 more
  1. 2022LAION-5B: An open large-scale dataset for training next generation image-text modelscited by 4,131
  2. 2019Generative Modeling by Estimating Gradients of the Data Distributioncited by 4,067
  3. 2021GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Modelscited by 3,595
  4. 2022Elucidating the Design Space of Diffusion-Based Generative Modelscited by 2,994
  5. 2015ShapeNet: An Information-Rich 3D Model Repositorycited by 2,880
  6. 2021NeuS: Learning Neural Implicit Surfaces by Volume Rendering for Multi-view Reconstructioncited by 1,701
  7. 2021Plenoxels: Radiance Fields without Neural Networkscited by 1,607
  8. 2015LSUN: Construction of a Large-scale Image Dataset using Deep Learning with Humans in the Loopcited by 1,350
  9. 2022TensoRF: Tensorial Radiance Fieldscited by 1,275
  10. 2020NeRF in the Wild: Neural Radiance Fields for Unconstrained Photo Collectionscited by 1,230
  11. 2021Efficient Geometry-aware 3D Generative Adversarial Networkscited by 1,204
  12. 2021Variational Diffusion Modelscited by 1,145
  13. 2020Improved Techniques for Training Score-Based Generative Modelscited by 1,036
  14. 2021Volume Rendering of Neural Implicit Surfacescited by 990
  15. 2021Direct Voxel Grid Optimization: Super-fast Convergence for Radiance Fields Reconstructioncited by 961
  16. 2020Neural Sparse Voxel Fieldscited by 959
  17. 2020GIRAFFE: Representing Scenes as Compositional Generative Neural Feature Fieldscited by 784
  18. 2020pi-GAN: Periodic Implicit Generative Adversarial Networks for 3D-Aware Image Synthesiscited by 708
  19. 2020GRAF: Generative Radiance Fields for 3D-Aware Image Synthesiscited by 688
  20. 2021UNISURF: Unifying Neural Implicit Surfaces and Radiance Fields for Multi-View Reconstructioncited by 643
  21. 2016Learning a Probabilistic Latent Space of Object Shapes via 3D Generative-Adversarial Modelingcited by 558
  22. 2021Zero-Shot Text-Guided Object Generation with Dream Fieldscited by 513
  23. 2021Label-Efficient Semantic Segmentation with Diffusion Modelscited by 495
  24. 2019PointFlow: 3D Point Cloud Generation with Continuous Normalizing Flowscited by 487
  25. 2018ABC: A Big CAD Model Dataset For Geometric Deep Learningcited by 339
  26. 2021Text2Mesh: Text-Driven Neural Stylization for Meshescited by 332
  27. 2022CLIP-Mesh: Generating textured meshes from text using pretrained image-text modelscited by 281
  28. 2020Learning Gradient Fields for Shape Generationcited by 245
  29. 2022Diffusion models as plug-and-play priorscited by 230
  30. 2018Pix3D: Dataset and Methods for Single-Image 3D Shape Modelingcited by 218
  31. 2020Image GANs meet Differentiable Rendering for Inverse Graphics and Interpretable 3D Neural Renderingcited by 101
  32. 2020Interacting particle solutions of Fokker-Planck equations through gradient-log-density estimationcited by 87
  33. 2022ReLU Fields: The Little Non-linearity That Couldcited by 87
  34. 2021FuseDream: Training-Free Text-to-Image Generation with Improved CLIP+GAN Space Optimizationcited by 71
  35. 2019HoloGAN: Unsupervised learning of 3D representations from natural imagescited by 59
  36. 2022Generative Multiplane Images: Making a 2D GAN 3D-Awarecited by 57
  37. 2022Understanding Pure CLIP Guidance for Voxel Grid NeRF Modelscited by 56
  38. 2021CAMPARI: Camera-Aware Decomposed Generative Neural Radiance Fieldscited by 55

50 most cited of 52 All 52 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 2; 2023: 189; 2024: 249; 2025: 115; 2026: 42.

597 papers.

Sort:

By area:

  1. 2023Zero-1-to-3: Zero-shot One Image to 3D Objectcited by 1,483
  2. 2023ProlificDreamer: High-Fidelity and Diverse Text-to-3D Generation with Variational Score Distillationcited by 1,207
  3. 2023MVDream: Multi-view Diffusion for 3D Generationcited by 928
  4. 2023DreamGaussian: Generative Gaussian Splatting for Efficient 3D Content Creationcited by 837
  5. 2023Objaverse-XL: A Universe of 10M+ 3D Objectscited by 767
  6. 2023Adversarial Diffusion Distillationcited by 741
  7. 2023Wonder3D: Single Image to 3D using Cross-Domain Diffusioncited by 682
  8. 2024Improved Distribution Matching Distillation for Fast Image Synthesiscited by 648
  9. 2023SyncDreamer: Generating Multiview-consistent Images from a Single-view Imagecited by 638
  10. 2023One-2-3-45: Any Single Image to 3D Mesh in 45 Seconds without Per-Shape Optimizationcited by 555
  11. 2024Latte: Latent Diffusion Transformer for Video Generationcited by 536
  12. 2023Instruct-NeRF2NeRF: Editing 3D Scenes with Instructionscited by 482
38 more
  1. 2024InstantMesh: Efficient 3D Mesh Generation from a Single Image with Sparse-view Large Reconstruction Modelscited by 421
  2. 2023Magic123: One Image to High-Quality 3D Object Generation Using Both 2D and 3D Diffusion Priorscited by 395
  3. 2023Shap-E: Generating Conditional 3D Implicit Functionscited by 379
  4. 2023Instant3D: Fast Text-to-3D with Sparse-View Generation and Large Reconstruction Modelcited by 376
  5. 2023Make-It-3D: High-Fidelity 3D Creation from A Single Image with Diffusion Priorcited by 341
  6. 2023ReconFusion: 3D Reconstruction with Diffusion Priorscited by 317
  7. 2023Text-to-3D using Gaussian Splattingcited by 299
  8. 2024GRM: Large Gaussian Reconstruction Model for Efficient 3D Reconstruction and Generationcited by 298
  9. 2023DreamLLM: Synergistic Multimodal Comprehension and Creationcited by 296
  10. 2023One-2-3-45++: Fast Single Image to 3D Objects with Consistent Multi-View Generation and 3D Diffusioncited by 283
  11. 2023Triplane Meets Gaussian Splatting: Fast and Generalizable Single-View 3D Reconstruction with Transformerscited by 257
  12. 2023ImageDream: Image-Prompt Multi-view Diffusion for 3D Generationcited by 256
  13. 2023GaussianDreamer: Fast Generation from Text to 3D Gaussians by Bridging 2D and 3D Diffusion Modelscited by 255
  14. 2023Text2Room: Extracting Textured 3D Meshes from 2D Text-to-Image Modelscited by 245
  15. 2023DreamBooth3D: Subject-Driven Text-to-3D Generationcited by 235
  16. 2024CRM: Single Image to 3D Textured Mesh with Convolutional Reconstruction Modelcited by 214
  17. 2023DMV3D: Denoising Multi-View Diffusion using 3D Large Reconstruction Modelcited by 213
  18. 20234D-fy: Text-to-4D Generation Using Hybrid Score Distillation Samplingcited by 184
  19. 2023TF-ICON: Diffusion-Based Training-Free Cross-Domain Image Compositioncited by 182
  20. 2023NerfDiff: Single-image View Synthesis with NeRF-guided Distillation from 3D-aware Diffusioncited by 179
  21. 2024Probing the 3D Awareness of Visual Foundation Modelscited by 175
  22. 2023Align Your Gaussians: Text-to-4D with Dynamic 3D Gaussians and Composed Diffusion Modelscited by 169
  23. 2023RichDreamer: A Generalizable Normal-Depth Diffusion Model for Detail Richness in Text-to-3Dcited by 168
  24. 2023DreamCraft3D: Hierarchical 3D Generation with Bootstrapped Diffusion Priorcited by 157
  25. 2024VD3D: Taming Large Video Diffusion Transformers for 3D Camera Controlcited by 156
  26. 2023DreamEditor: Text-Driven 3D Scene Editing with Neural Fieldscited by 154
  27. 2022Dream3D: Zero-Shot Text-to-3D Synthesis Using 3D Shape Prior and Text-to-Image Diffusion Modelscited by 151
  28. 2024GPT-4V(ision) is a Human-Aligned Evaluator for Text-to-3D Generationcited by 151
  29. 2024AC3D: Analyzing and Improving 3D Camera Control in Video Diffusion Transformerscited by 151
  30. 2023Re-imagine the Negative Prompt Algorithm: Transform 2D Diffusion into 3D, alleviate Janus problem and Beyondcited by 139
  31. 2023SweetDreamer: Aligning Geometric Priors in 2D Diffusion for Consistent Text-to-3Dcited by 134
  32. 2023Let 2D Diffusion Model Know 3D-Consistency for Robust Text-to-3D Generationcited by 133
  33. 2024Unique3D: High-Quality and Efficient 3D Mesh Generation from a Single Imagecited by 132
  34. 2023Text-to-3D with Classifier Score Distillationcited by 127
  35. 2023State of the Art on Diffusion Models for Visual Computingcited by 126
  36. 2023Consistent4D: Consistent 360deg Dynamic Object Generation from Monocular Videocited by 124
  37. 2023Text2NeRF: Text-Driven 3D Scene Generation with Neural Radiance Fieldscited by 122
  38. 20234DGen: Grounded 4D Content Generation with Spatial-temporal Consistencycited by 117

50 most cited of 597 All 597 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2022RenderDiffusion: Image Diffusion for 3D Reconstruction, Inpainting and Generation
  2. 2023HoloDiffusion: Training a 3D Diffusion Model using 2D Images
  3. 2024Diffusion2: Dynamic 3D Content Generation via Score Composition of Video and Multi-view Diffusion Models
  4. 2023Breathing New Life into 3D Assets with Generative Repainting
  5. 2022Let us Build Bridges: Understanding and Extending Diffusion Generative Models
  6. 2024A Lesson in Splats: Teacher-Guided Diffusion for 3D Gaussian Splats Generation with 2D Supervision
  7. 2021Conditional Image Generation with Score-Based Diffusion Models
  8. 2024DPBridge: Latent Diffusion Bridge for Dense Prediction