mlsanityData to 14 Oct 2026

Light Transport-aware Diffusion Posterior Sampling for Single-View Reconstruction of 3D Volumes

arXiv 2501.05226 ↗First posted 9 Jan 2025cs.CV cs.LGcited by 0

On the map: Diffusion models → 3D, point cloud and scene generation

Abstract (arXiv)

We introduce a single-view reconstruction technique of volumetric fields in which multiple light scattering effects are omnipresent, such as in clouds. We model the unknown distribution of volumetric fields using an unconditional diffusion model trained on a novel benchmark dataset comprising 1,000 synthetically simulated volumetric density fields. The neural diffusion model is trained on the latent codes of a novel, diffusion-friendly, monoplanar representation. The generative model is used to incorporate a tailored parametric diffusion posterior sampling technique into different reconstruction tasks. A physically-based differentiable volume renderer is employed to provide gradients with respect to light transport in the latent space. This stands in contrast to classic NeRF approaches and makes the reconstructions better aligned with observed data. Through various experiments, we demonstrate single-view reconstruction of volumetric clouds at a previously unattainable quality.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in 3D, point cloud and scene generation, one of 14 directions in Diffusion models. Builds on 69 of its 88 references that are in the map. First posted on arXiv 9 Jan 2025.

Builds on

69 of 88 references.

Closest ancestors · 22

Same direction, newest first

  1. 2023Wonder3D: Single Image to 3D using Cross-Domain Diffusion23 Oct
  2. 2023HoloFusion: Towards Photo-realistic 3D Generative Modeling28 Aug
  3. 2023AutoDecoding Latent 3D Diffusion Models7 Jul
  4. 2023Viewset Diffusion: (0-)Image-Conditioned 3D Generative Models from 2D Data13 Jun
  5. 2023NeuralField-LDM: Scene Generation with Hierarchical Latent Diffusion Models19 Apr
  6. 2023Single-Stage Diffusion NeRF: A Unified Approach to 3D Generation and Reconstruction13 Apr
  7. 2023Generative Novel View Synthesis with 3D-Aware Diffusion Models5 Apr
  8. 2023HoloDiffusion: Training a 3D Diffusion Model using 2D Images29 Mar
  9. 2023Zero-1-to-3: Zero-shot One Image to 3D Object20 Mar
  10. 20233DGen: Triplane Latent Diffusion for Textured Mesh Generation9 Mar
  11. 20233DShape2VecSet: A 3D Shape Representation for Neural Fields and Generative Diffusion Models26 Jan
  12. 2022Point-E: A System for Generating 3D Point Clouds from Complex Prompts16 Dec
10 more
  1. 2022Rodin: A Generative Model for Sculpting 3D Digital Avatars Using Diffusion12 Dec
  2. 2022DiffRF: Rendering-Guided 3D Radiance Field Diffusion2 Dec
  3. 20223D Neural Field Generation using Triplane Diffusion30 Nov
  4. 20223DDesigner: Towards Photorealistic 3D Object Generation and Editing with Text-guided Diffusion Models25 Nov
  5. 2022Diffusion-SDF: Conditional Generative Modeling of Signed Distance Functions24 Nov
  6. 2022RenderDiffusion: Image Diffusion for 3D Reconstruction, Inpainting and Generation17 Nov
  7. 2022LION: Latent Point Diffusion Models for 3D Shape Generation12 Oct
  8. 2022Novel View Synthesis with Diffusion Models6 Oct
  9. 2022DreamFusion: Text-to-3D using 2D Diffusion29 Sep
  10. 2021Diffusion Probabilistic Models for 3D Point Cloud Generation2 Mar

Other references · 47

  1. 2020Denoising Diffusion Probabilistic Modelscited by 21,076
  2. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,305
  3. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,863
  4. 2018The Unreasonable Effectiveness of Deep Features as a Perceptual Metriccited by 11,177
  5. 2020Denoising Diffusion Implicit Modelscited by 9,781
  6. 2020Score-Based Generative Modeling through Stochastic Differential Equationscited by 9,122
  7. 2015Deep Unsupervised Learning using Nonequilibrium Thermodynamicscited by 7,058
  8. 2023SDXL: Improving Latent Diffusion Models for High-Resolution Image Synthesiscited by 4,356
  9. 2019Generative Modeling by Estimating Gradients of the Data Distributioncited by 4,101
  10. 2022Instant Neural Graphics Primitives with a Multiresolution Hash Encodingcited by 3,723
  11. 2021Plenoxels: Radiance Fields without Neural Networkscited by 1,613
  12. 2020pixelNeRF: Neural Radiance Fields from One or Few Imagescited by 1,584
35 more
  1. 2022Diffusion Posterior Sampling for General Noisy Inverse Problemscited by 1,416
  2. 2022TensoRF: Tensorial Radiance Fieldscited by 1,280
  3. 2020Improved Techniques for Training Score-Based Generative Modelscited by 1,043
  4. 2021PlenOctrees for Real-time Rendering of Neural Radiance Fieldscited by 905
  5. 2019Scene Representation Networks: Continuous 3D-Structure-Aware Neural Scene Representationscited by 864
  6. 2020GIRAFFE: Representing Scenes as Compositional Generative Neural Feature Fieldscited by 785
  7. 2022Block-NeRF: Scalable Large Scene Neural View Synthesiscited by 724
  8. 2020pi-GAN: Periodic Implicit Generative Adversarial Networks for 3D-Aware Image Synthesiscited by 708
  9. 2021IBRNet: Learning Multi-View Image-Based Renderingcited by 703
  10. 2020GRAF: Generative Radiance Fields for 3D-Aware Image Synthesiscited by 689
  11. 2020AdaBins: Depth Estimation using Adaptive Binscited by 601
  12. 20213D Shape Generation and Completion through Point-Voxel Diffusioncited by 516
  13. 2021StyleNeRF: A Style-based 3D-Aware Generator for High-resolution Image Synthesiscited by 477
  14. 2022GET3D: A Generative Model of High Quality 3D Textured Shapes Learned from Imagescited by 446
  15. 2021Mega-NeRF: Scalable Construction of Large-Scale NeRFs for Virtual Fly-Throughscited by 398
  16. 2021Advances in Neural Renderingcited by 365
  17. 2023RealFusion: 360deg Reconstruction of Any Object from a Single Imagecited by 306
  18. 2022SDFusion: Multimodal 3D Shape Completion, Reconstruction, and Generationcited by 288
  19. 2023MeshDiffusion: Score-based Generative 3D Mesh Modelingcited by 199
  20. 2023Solving Inverse Problems with Latent Diffusion Models via Hard Data Consistencycited by 182
  21. 2023HyperDiffusion: Generating Implicit Neural Fields with Weight-Space Diffusioncited by 152
  22. 2021CIPS-3D: A 3D-Aware Generator of GANs Based on Conditionally-Independent Pixel Synthesiscited by 149
  23. 2022GAUDI: A Neural Architect for Immersive 3D Scene Generationcited by 136
  24. 2023State of the Art on Diffusion Models for Visual Computingcited by 127
  25. 2020Differentiable Rendering: A Surveycited by 123
  26. 2023Diffusion with Forward Models: Solving Stochastic Inverse Problems Without Direct Supervisioncited by 112
  27. 2022Vision Transformer for NeRF-Based View Synthesis from a Single Input Imagecited by 104
  28. 2021Compressive Neural Representations of Volumetric Scalar Fieldscited by 64
  29. 2023Make-A-Volume: Leveraging Latent Diffusion Models for Cross-Modality 3D Brain MRI Synthesiscited by 43
  30. 2022TORE: Token Reduction for Efficient Human Mesh Recovery with Transformercited by 41
  31. 2023DISTWAR: Fast Differentiable Rendering on Raster-based Rendering Pipelinescited by 36
  32. 2021Fast Neural Representations for Direct Volume Renderingcited by 32
  33. 2021Global Transport for Fluid Reconstruction with Learned Self-Supervisioncited by 18
  34. 2021Differentiable Direct Volume Renderingcited by 15
  35. 2022GenDR: A Generalized Differentiable Renderercited by 12

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025NnD: Diffusion-based Generation of Physically-Nonnegative Objects
  2. 2025Consistency Diffusion Models for Single-Image 3D Reconstruction with Priors
  3. 2025Splatent: Splatting Diffusion Latents for Novel View Synthesis
  4. 2024Diffusion-based Light Field Synthesis
  5. 2023Light Field Diffusion for Single-View Novel View Synthesis
  6. 2025Yesnt: Are Diffusion Relighting Models Ready for Capture Stage Compositing? A Hybrid Alternative to Bridge the Gap
  7. 2026Three-Step Conditional Diffusion 3D Reconstruction for Light-Field Microscopy
  8. 2026Predicting 3D structure by latent posterior sampling