# EnvGS: Modeling View-Dependent Appearance with Environment Gaussian

arXiv: https://arxiv.org/abs/2412.15215 · HTML: https://mlsanity.com/paper/2412.15215/

## Key facts

- Cited by 35 papers in the map (data through 14 Oct 2026).
- Builds on 37 of its 53 references that are in the map.
- First posted on arXiv 19 Dec 2024.

## Builds on (most cited first, 30 of 37)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [Very Deep Convolutional Networks for Large-Scale Image Recognition](https://mlsanity.com/paper/1409.1556/) (2014)
- [The Unreasonable Effectiveness of Deep Features as a Perceptual Metric](https://mlsanity.com/paper/1801.03924/) (2018)
- [3D Gaussian Splatting for Real-Time Radiance Field Rendering](https://mlsanity.com/paper/2308.04079/) (2023)
- [Mip-NeRF 360: Unbounded Anti-Aliased Neural Radiance Fields](https://mlsanity.com/paper/2111.12077/) (2021)
- [Mip-NeRF: A Multiscale Representation for Anti-Aliasing Neural Radiance Fields](https://mlsanity.com/paper/2103.13415/) (2021)
- [NeuS: Learning Neural Implicit Surfaces by Volume Rendering for Multi-view Reconstruction](https://mlsanity.com/paper/2106.10689/) (2021)
- [Plenoxels: Radiance Fields without Neural Networks](https://mlsanity.com/paper/2112.05131/) (2021)
- [TensoRF: Tensorial Radiance Fields](https://mlsanity.com/paper/2203.09517/) (2022)
- [2D Gaussian Splatting for Geometrically Accurate Radiance Fields](https://mlsanity.com/paper/2403.17888/) (2024)
- [Direct Voxel Grid Optimization: Super-fast Convergence for Radiance Fields Reconstruction](https://mlsanity.com/paper/2111.11215/) (2021)
- [Neural Sparse Voxel Fields](https://mlsanity.com/paper/2007.11571/) (2020)
- [Scaffold-GS: Structured 3D Gaussians for View-Adaptive Rendering](https://mlsanity.com/paper/2312.00109/) (2023)
- [Zip-NeRF: Anti-Aliased Grid-Based Neural Radiance Fields](https://mlsanity.com/paper/2304.06706/) (2023)
- [Mip-Splatting: Alias-free 3D Gaussian Splatting](https://mlsanity.com/paper/2311.16493/) (2023)
- [Ref-NeRF: Structured View-Dependent Appearance for Neural Radiance Fields](https://mlsanity.com/paper/2112.03907/) (2021)
- [Neuralangelo: High-Fidelity Neural Surface Reconstruction](https://mlsanity.com/paper/2306.03092/) (2023)
- [NeuS2: Fast Learning of Neural Implicit Surfaces for Multi-view Reconstruction](https://mlsanity.com/paper/2212.05231/) (2022)
- [GaussianShader: 3D Gaussian Splatting with Shading Functions for Reflective Surfaces](https://mlsanity.com/paper/2311.17977/) (2023)
- [Lotus: Diffusion-based Visual Foundation Model for High-quality Dense Prediction](https://mlsanity.com/paper/2409.18124/) (2024)
- [NeRO: Neural Geometry and BRDF Reconstruction of Reflective Objects from Multiview Images](https://mlsanity.com/paper/2305.17398/) (2023)
- [StableNormal: Reducing Diffusion Variance for Stable and Sharp Normal](https://mlsanity.com/paper/2406.16864/) (2024)
- [3D Gaussian Ray Tracing: Fast Tracing of Particle Scenes](https://mlsanity.com/paper/2407.07090/) (2024)
- [Rethinking Inductive Biases for Surface Normal Estimation](https://mlsanity.com/paper/2403.00712/) (2024)
- [Prompting Depth Anything for 4K Resolution Accurate Metric Depth Estimation](https://mlsanity.com/paper/2412.14015/) (2024)
- [Detector-Free Structure from Motion](https://mlsanity.com/paper/2306.15669/) (2023)
- [ENVIDR: Implicit Differentiable Renderer with Neural Environment Lighting](https://mlsanity.com/paper/2303.13022/) (2023)
- [GS^3: Efficient Relighting with Triple Gaussian Splatting](https://mlsanity.com/paper/2410.11419/) (2024)
- [NeRF-Casting: Improved View-Dependent Appearance with Consistent Reflections](https://mlsanity.com/paper/2405.14871/) (2024)

## Built on by (most cited first, 30 of 35)

- [MaterialRefGS: Reflective Gaussian Splatting with Multi-view Consistent Material Inference](https://mlsanity.com/paper/2510.11387/) (2025)
- [Spherical Voronoi: Directional Appearance as a Differentiable Partition of the Sphere](https://mlsanity.com/paper/2512.14180/) (2025)
- [MeshSplats: Mesh-Based Rendering with Gaussian Splatting Initialization](https://mlsanity.com/paper/2502.07754/) (2025)
- [Decomposing Densification in Gaussian Splatting for Faster 3D Scene Reconstruction](https://mlsanity.com/paper/2507.20239/) (2025)
- [RGS-DR: Deferred Reflections and Residual Shading in 2D Gaussian Splatting](https://mlsanity.com/paper/2504.18468/) (2025)
- [MADrive: Memory-Augmented Driving Scene Modeling](https://mlsanity.com/paper/2506.21520/) (2025)
- [Radiometrically Consistent Gaussian Surfels for Inverse Rendering](https://mlsanity.com/paper/2603.01491/) (2026)
- [RT-Splatting: Joint Reflection-Transmission Modeling with Gaussian Splatting](https://mlsanity.com/paper/2605.18263/) (2026)
- [REdiSplats: Ray Tracing for Editable Gaussian Splatting](https://mlsanity.com/paper/2503.12284/) (2025)
- [LidarPainter: One-Step Away From Any Lidar View To Novel Guidance](https://mlsanity.com/paper/2507.12114/) (2025)
- [TR-Gaussians: High-fidelity Real-time Rendering of Planar Transmission and Reflection with 3D Gaussian Splatting](https://mlsanity.com/paper/2511.13009/) (2025)
- [GTAvatar: Bridging Gaussian Splatting and Texture Mapping for Relightable and Editable Gaussian Avatars](https://mlsanity.com/paper/2512.09162/) (2025)
- [EAG-PT: Emission-Aware Gaussians and Path Tracing for Diffuse Indoor Scene Reconstruction and Editing](https://mlsanity.com/paper/2601.23065/) (2026)
- [RT-GS: Gaussian Splatting with Reflection and Transmittance Primitives](https://mlsanity.com/paper/2604.00509/) (2026)
- [LumiMotion: Improving Gaussian Relighting with Scene Dynamics](https://mlsanity.com/paper/2604.10994/) (2026)
- [TransmissiveGS: Residual-Guided Disentangled Gaussian Splatting for Transmissive Scene Reconstruction and Rendering](https://mlsanity.com/paper/2605.10705/) (2026)
- [Revisiting Photometric Ambiguity for Accurate Gaussian-Splatting Surface Reconstruction](https://mlsanity.com/paper/2605.12494/) (2026)
- [MoGaFace: Momentum-Guided and Texture-Aware Gaussian Avatars for Consistent Facial Geometry](https://mlsanity.com/paper/2508.01218/) (2025)
- [ARGS: Advanced Regularization on Aligning Gaussians over the Surface](https://mlsanity.com/paper/2508.21344/) (2025)
- [UTrice: Unifying Primitives in Differentiable Ray Tracing and Rasterization via Triangles for Particle-Based 3D Scenes](https://mlsanity.com/paper/2512.04421/) (2025)
- [HybridSplat: Fast Reflection-baked Gaussian Tracing using Hybrid Splatting](https://mlsanity.com/paper/2512.08334/) (2025)
- [TraceFlow: Dynamic 3D Reconstruction of Specular Scenes Driven by Ray Tracing](https://mlsanity.com/paper/2512.10095/) (2025)
- [Beyond a Single Light: A Large-Scale Aerial Dataset for Urban Scene Reconstruction Under Varying Illumination](https://mlsanity.com/paper/2512.14200/) (2025)
- [Augmented Radiance Field: A General Framework for Enhanced Gaussian Splatting](https://mlsanity.com/paper/2602.19916/) (2026)
- [SSR-GS: Separating Specular Reflection in Gaussian Splatting for Glossy Surface Reconstruction](https://mlsanity.com/paper/2603.05152/) (2026)
- [Ref-DGS: Reflective Dual Gaussian Splatting](https://mlsanity.com/paper/2603.07664/) (2026)
- [S2D: Sparse to Dense Lifting for 3D Reconstruction with Minimal Inputs](https://mlsanity.com/paper/2603.10893/) (2026)
- [GLINT: Modeling Scene-Scale Transparency via Gaussian Radiance Transport](https://mlsanity.com/paper/2603.26181/) (2026)
- [Differentiable Ray Tracing with Gaussians for Unified Radio Propagation Simulation and View Synthesis](https://mlsanity.com/paper/2605.07781/) (2026)
- [Plenoptic Condensation: A Novel Approach to Generalized Scene Reconstruction](https://mlsanity.com/paper/2607.18151/) (2026)

## Abstract (arXiv)

Reconstructing complex reflections in real-world scenes from 2D images is essential for achieving photorealistic novel view synthesis. Existing methods that utilize environment maps to model reflections from distant lighting often struggle with high-frequency reflection details and fail to account for near-field reflections. In this work, we introduce EnvGS, a novel approach that employs a set of Gaussian primitives as an explicit 3D representation for capturing reflections of environments. These environment Gaussian primitives are incorporated with base Gaussian primitives to model the appearance of the whole scene. To efficiently render these environment Gaussian primitives, we developed a ray-tracing-based renderer that leverages the GPU's RT core for fast rendering. This allows us to jointly optimize our model for high-quality reconstruction while maintaining real-time rendering speeds. Results from multiple real-world and synthetic datasets demonstrate that our method produces significantly more detailed reflections, achieving the best rendering quality in real-time novel view synthesis. The code is available at https://zju3dv.github.io/envgs.