# Relightable 3D Gaussians: Realistic Point Cloud Relighting with BRDF Decomposition and Ray Tracing

arXiv: https://arxiv.org/abs/2311.16043 · HTML: https://mlsanity.com/paper/2311.16043/

## Key facts

- Cited by 142 papers in the map (data through 14 Oct 2026).
- Builds on 43 of its 55 references that are in the map.
- First posted on arXiv 27 Nov 2023.

## Builds on (most cited first, 30 of 43)

- [3D Gaussian Splatting for Real-Time Radiance Field Rendering](https://mlsanity.com/paper/2308.04079/) (2023)
- [Instant Neural Graphics Primitives with a Multiresolution Hash Encoding](https://mlsanity.com/paper/2201.05989/) (2022)
- [Mip-NeRF 360: Unbounded Anti-Aliased Neural Radiance Fields](https://mlsanity.com/paper/2111.12077/) (2021)
- [Mip-NeRF: A Multiscale Representation for Anti-Aliasing Neural Radiance Fields](https://mlsanity.com/paper/2103.13415/) (2021)
- [NeuS: Learning Neural Implicit Surfaces by Volume Rendering for Multi-view Reconstruction](https://mlsanity.com/paper/2106.10689/) (2021)
- [Plenoxels: Radiance Fields without Neural Networks](https://mlsanity.com/paper/2112.05131/) (2021)
- [TensoRF: Tensorial Radiance Fields](https://mlsanity.com/paper/2203.09517/) (2022)
- [Volume Rendering of Neural Implicit Surfaces](https://mlsanity.com/paper/2106.12052/) (2021)
- [Depth-supervised NeRF: Fewer Views and Faster Training for Free](https://mlsanity.com/paper/2107.02791/) (2021)
- [KiloNeRF: Speeding up Neural Radiance Fields with Thousands of Tiny MLPs](https://mlsanity.com/paper/2103.13744/) (2021)
- [Zip-NeRF: Anti-Aliased Grid-Based Neural Radiance Fields](https://mlsanity.com/paper/2304.06706/) (2023)
- [SuGaR: Surface-Aligned Gaussian Splatting for Efficient 3D Mesh Reconstruction and High-Quality Mesh Rendering](https://mlsanity.com/paper/2311.12775/) (2023)
- [Ref-NeRF: Structured View-Dependent Appearance for Neural Radiance Fields](https://mlsanity.com/paper/2112.03907/) (2021)
- [FastNeRF: High-Fidelity Neural Rendering at 200FPS](https://mlsanity.com/paper/2103.10380/) (2021)
- [NeRV: Neural Reflectance and Visibility Fields for Relighting and View Synthesis](https://mlsanity.com/paper/2012.03927/) (2020)
- [MonoSDF: Exploring Monocular Geometric Cues for Neural Implicit Surface Reconstruction](https://mlsanity.com/paper/2206.00665/) (2022)
- [Baking Neural Radiance Fields for Real-Time View Synthesis](https://mlsanity.com/paper/2103.14645/) (2021)
- [NeRD: Neural Reflectance Decomposition from Image Collections](https://mlsanity.com/paper/2012.03918/) (2020)
- [Extracting Triangular 3D Models, Materials, and Lighting From Images](https://mlsanity.com/paper/2111.12503/) (2021)
- [PhySG: Inverse Rendering with Spherical Gaussians for Physics-based Material Editing and Relighting](https://mlsanity.com/paper/2104.00674/) (2021)
- [Neural Point-Based Graphics](https://mlsanity.com/paper/1906.08240/) (2019)
- [MobileNeRF: Exploiting the Polygon Rasterization Pipeline for Efficient Neural Field Rendering on Mobile Architectures](https://mlsanity.com/paper/2208.00277/) (2022)
- [Differentiable Surface Splatting for Point-based Geometry Processing](https://mlsanity.com/paper/1906.04173/) (2019)
- [Neural Reflectance Fields for Appearance Acquisition](https://mlsanity.com/paper/2008.03824/) (2020)
- [Modeling Indirect Illumination for Inverse Rendering](https://mlsanity.com/paper/2204.06837/) (2022)
- [TensoIR: Tensorial Inverse Rendering](https://mlsanity.com/paper/2304.12461/) (2023)
- [Shape, Light, and Material Decomposition from Images using Monte Carlo Rendering and Denoising](https://mlsanity.com/paper/2206.03380/) (2022)
- [NeRO: Neural Geometry and BRDF Reconstruction of Reflective Objects from Multiview Images](https://mlsanity.com/paper/2305.17398/) (2023)
- [IRON: Inverse Rendering by Optimizing Neural SDFs and Materials from Photometric Images](https://mlsanity.com/paper/2204.02232/) (2022)
- [EfficientNeRF: Efficient Neural Radiance Fields](https://mlsanity.com/paper/2206.00878/) (2022)

## Built on by (most cited first, 30 of 142)

- [GIR: 3D Gaussian Inverse Rendering for Relightable Scene Factorization](https://mlsanity.com/paper/2312.05133/) (2023)
- [3D Gaussian Splatting: Survey, Technologies, Challenges, and Opportunities](https://mlsanity.com/paper/2407.17418/) (2024)
- [BAGS: Blur Agnostic Gaussian Splatting through Multi-Scale Kernel Modeling](https://mlsanity.com/paper/2403.04926/) (2024)
- [Gaussian Splatting: 3D Reconstruction and Novel View Synthesis, a Review](https://mlsanity.com/paper/2405.03417/) (2024)
- [Matrix3D: Large Photogrammetry Model All-in-One](https://mlsanity.com/paper/2502.07685/) (2025)
- [Ref-GS: Directional Factorization for 2D Gaussian Splatting](https://mlsanity.com/paper/2412.00905/) (2024)
- [GauStudio: A Modular Framework for 3D Gaussian Splatting and Beyond](https://mlsanity.com/paper/2403.19632/) (2024)
- [ScribbleLight: Single Image Indoor Relighting with Scribbles](https://mlsanity.com/paper/2411.17696/) (2024)
- [SVG-IR: Spatially-Varying Gaussian Splatting for Inverse Rendering](https://mlsanity.com/paper/2504.06815/) (2025)
- [MaterialRefGS: Reflective Gaussian Splatting with Multi-view Consistent Material Inference](https://mlsanity.com/paper/2510.11387/) (2025)
- [PIN-WM: Learning Physics-INformed World Models for Non-Prehensile Manipulation](https://mlsanity.com/paper/2504.16693/) (2025)
- [GUAVA: Generalizable Upper Body 3D Gaussian Avatar](https://mlsanity.com/paper/2505.03351/) (2025)
- [Z-Splat: Z-Axis Gaussian Splatting for Camera-Sonar Fusion](https://mlsanity.com/paper/2404.04687/) (2024)
- [Deformable Radial Kernel Splatting](https://mlsanity.com/paper/2412.11752/) (2024)
- [GeoSVR: Taming Sparse Voxels for Geometrically Accurate Surface Reconstruction](https://mlsanity.com/paper/2509.18090/) (2025)
- [TransparentGS: Fast Inverse Rendering of Transparent Objects with Gaussians](https://mlsanity.com/paper/2504.18768/) (2025)
- [Unifying Appearance Codes and Bilateral Grids for Driving Scene Gaussian Splatting](https://mlsanity.com/paper/2506.05280/) (2025)
- [ReCap: Better Gaussian Relighting with Cross-Environment Captures](https://mlsanity.com/paper/2412.07534/) (2024)
- [Code as Worlds: Agentic Discovery of Executable World Representations for Physical Reasoning](https://mlsanity.com/paper/2608.27549/) (2026)
- [RainyGS: Efficient Rain Synthesis with Physically-Based Gaussian Splatting](https://mlsanity.com/paper/2503.21442/) (2025)
- [Gaussian Splatting with Discretized SDF for Relightable Assets](https://mlsanity.com/paper/2507.15629/) (2025)
- [Leveraging Thermal Modality to Enhance Reconstruction in Low-Light Conditions](https://mlsanity.com/paper/2403.14053/) (2024)
- [GS-ROR2: Bidirectional-guided 3DGS and SDF for Reflective Object Relighting and Reconstruction](https://mlsanity.com/paper/2406.18544/) (2024)
- [GUS-IR: Gaussian Splatting with Unified Shading for Inverse Rendering](https://mlsanity.com/paper/2411.07478/) (2024)
- [SweepEvGS: Event-Based 3D Gaussian Splatting for Macro and Micro Radiance Field Rendering from a Single Sweep](https://mlsanity.com/paper/2412.11579/) (2024)
- [RTR-GS: 3D Gaussian Splatting for Inverse Rendering with Radiance Transfer and Reflection](https://mlsanity.com/paper/2507.07733/) (2025)
- [Generative World Renderer](https://mlsanity.com/paper/2604.02329/) (2026)
- [TexGS-VolVis: Expressive Scene Editing for Volume Visualization via Textured Gaussian Splatting](https://mlsanity.com/paper/2507.13586/) (2025)
- [LuxRemix: Lighting Decomposition and Remixing for Indoor Scenes](https://mlsanity.com/paper/2601.15283/) (2026)
- [MV-CoLight: Efficient Object Compositing with Consistent Lighting and Shadow Generation](https://mlsanity.com/paper/2505.21483/) (2025)

## Abstract (arXiv)

In this paper, we present a novel differentiable point-based rendering framework to achieve photo-realistic relighting. To make the reconstructed scene relightable, we enhance vanilla 3D Gaussians by associating extra properties, including normal vectors, BRDF parameters, and incident lighting from various directions. From a collection of multi-view images, the 3D scene is optimized through 3D Gaussian Splatting while BRDF and lighting are decomposed by physically based differentiable rendering. To produce plausible shadow effects in photo-realistic relighting, we introduce an innovative point-based ray tracing with the bounding volume hierarchies for efficient visibility pre-computation. Extensive experiments demonstrate our improved BRDF estimation, novel view synthesis and relighting results compared to state-of-the-art approaches. The proposed framework showcases the potential to revolutionize the mesh-based graphics pipeline with a point-based pipeline enabling editing, tracing, and relighting.