# Gaussian Splatting with NeRF-based Color and Opacity

arXiv: https://arxiv.org/abs/2312.13729 · HTML: https://mlsanity.com/paper/2312.13729/

## Key facts

- Cited by 30 papers in the map (data through 14 Oct 2026).
- Builds on 28 of its 38 references that are in the map.
- First posted on arXiv 21 Dec 2023.

## Builds on (most cited first, 28 of 28)

- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [3D Gaussian Splatting for Real-Time Radiance Field Rendering](https://mlsanity.com/paper/2308.04079/) (2023)
- [Instant Neural Graphics Primitives with a Multiresolution Hash Encoding](https://mlsanity.com/paper/2201.05989/) (2022)
- [Mip-NeRF 360: Unbounded Anti-Aliased Neural Radiance Fields](https://mlsanity.com/paper/2111.12077/) (2021)
- [Mip-NeRF: A Multiscale Representation for Anti-Aliasing Neural Radiance Fields](https://mlsanity.com/paper/2103.13415/) (2021)
- [Plenoxels: Radiance Fields without Neural Networks](https://mlsanity.com/paper/2112.05131/) (2021)
- [TensoRF: Tensorial Radiance Fields](https://mlsanity.com/paper/2203.09517/) (2022)
- [Efficient Geometry-aware 3D Generative Adversarial Networks](https://mlsanity.com/paper/2112.07945/) (2021)
- [4D Gaussian Splatting for Real-Time Dynamic Scene Rendering](https://mlsanity.com/paper/2310.08528/) (2023)
- [Direct Voxel Grid Optimization: Super-fast Convergence for Radiance Fields Reconstruction](https://mlsanity.com/paper/2111.11215/) (2021)
- [Neural Sparse Voxel Fields](https://mlsanity.com/paper/2007.11571/) (2020)
- [Depth-supervised NeRF: Fewer Views and Faster Training for Free](https://mlsanity.com/paper/2107.02791/) (2021)
- [Block-NeRF: Scalable Large Scene Neural View Synthesis](https://mlsanity.com/paper/2202.05263/) (2022)
- [RegNeRF: Regularizing Neural Radiance Fields for View Synthesis from Sparse Inputs](https://mlsanity.com/paper/2112.00724/) (2021)
- [3D-R2N2: A Unified Approach for Single and Multi-view 3D Object Reconstruction](https://mlsanity.com/paper/1604.00449/) (2016)
- [Ref-NeRF: Structured View-Dependent Appearance for Neural Radiance Fields](https://mlsanity.com/paper/2112.03907/) (2021)
- [Extracting Triangular 3D Models, Materials, and Lighting From Images](https://mlsanity.com/paper/2111.12503/) (2021)
- [Neural RGB-D Surface Reconstruction](https://mlsanity.com/paper/2104.04532/) (2021)
- [Dense Depth Priors for Neural Radiance Fields from Sparse Input Views](https://mlsanity.com/paper/2112.03288/) (2021)
- [Single-Stage Diffusion NeRF: A Unified Approach to 3D Generation and Reconstruction](https://mlsanity.com/paper/2304.06714/) (2023)
- [Shape, Light, and Material Decomposition from Images using Monte Carlo Rendering and Denoising](https://mlsanity.com/paper/2206.03380/) (2022)
- [NeRO: Neural Geometry and BRDF Reconstruction of Reflective Objects from Multiview Images](https://mlsanity.com/paper/2305.17398/) (2023)
- [Hierarchical Surface Prediction for 3D Object Reconstruction](https://mlsanity.com/paper/1704.00710/) (2017)
- [ENVIDR: Implicit Differentiable Renderer with Neural Environment Lighting](https://mlsanity.com/paper/2303.13022/) (2023)
- [Flexible Techniques for Differentiable Rendering with 3D Gaussians](https://mlsanity.com/paper/2308.14737/) (2023)
- [Hypernetwork approach to generating point clouds](https://mlsanity.com/paper/2003.00802/) (2020)
- [Continuous Conditional Random Field Convolution for Point Cloud Segmentation](https://mlsanity.com/paper/2110.06085/) (2021)
- [MultiPlaneNeRF: Neural Radiance Field with Non-Trainable Representation](https://mlsanity.com/paper/2305.10579/) (2023)

## Built on by (most cited first, 30 of 30)

- [Recent Advances in 3D Gaussian Splatting](https://mlsanity.com/paper/2403.11134/) (2024)
- [3D Gaussian Splatting: Survey, Technologies, Challenges, and Opportunities](https://mlsanity.com/paper/2407.17418/) (2024)
- [Spec-Gaussian: Anisotropic View-Dependent Appearance for 3D Gaussian Splatting](https://mlsanity.com/paper/2402.15870/) (2024)
- [GS^3: Efficient Relighting with Triple Gaussian Splatting](https://mlsanity.com/paper/2410.11419/) (2024)
- [D-MiSo: Editing Dynamic 3D Scenes using Multi-Gaussians Soup](https://mlsanity.com/paper/2405.14276/) (2024)
- [NeRF Is a Valuable Assistant for 3D Gaussian Splatting](https://mlsanity.com/paper/2507.23374/) (2025)
- [HyperGS: Hyperspectral 3D Gaussian Splatting](https://mlsanity.com/paper/2412.12849/) (2024)
- [Query3D: LLM-Powered Open-Vocabulary Scene Segmentation with Language Embedded 3D Gaussian](https://mlsanity.com/paper/2408.03516/) (2024)
- [Content-Aware Texturing for Gaussian Splatting](https://mlsanity.com/paper/2512.02621/) (2025)
- [Dropping Anchor and Spherical Harmonics for Sparse-view Gaussian Splatting](https://mlsanity.com/paper/2602.20933/) (2026)
- [PR-ENDO: Physically Based Relightable Gaussian Splatting for Endoscopy](https://mlsanity.com/paper/2411.12510/) (2024)
- [Reflections Unlock: Geometry-Aware Reflection Disentanglement in 3D Gaussian Splatting for Photorealistic Scenes Rendering](https://mlsanity.com/paper/2507.06103/) (2025)
- [Affine-Equivariant Kernel Space Encoding for NeRF Editing](https://mlsanity.com/paper/2508.02831/) (2025)
- [RefGaussian: Disentangling Reflections from 3D Gaussian Splatting for Realistic Rendering](https://mlsanity.com/paper/2406.05852/) (2024)
- [ImViD: Immersive Volumetric Videos for Enhanced VR Engagement](https://mlsanity.com/paper/2503.14359/) (2025)
- [Leveraging NeRF-Rendered Images for 3D Gaussian Splatting](https://mlsanity.com/paper/2606.09034/) (2026)
- [TSGaussian: Semantic and Depth-Guided Target-Specific Gaussian Splatting from Sparse Views](https://mlsanity.com/paper/2412.10051/) (2024)
- [HuSc3D: Human Sculpture dataset for 3D object reconstruction](https://mlsanity.com/paper/2506.07628/) (2025)
- [Realizing Immersive Volumetric Video: A Multimodal Framework for 6-DoF VR Engagement](https://mlsanity.com/paper/2604.09473/) (2026)
- [GPN: Generative Point-based NeRF](https://mlsanity.com/paper/2404.08312/) (2024)
- [A Refined 3D Gaussian Representation for High-Quality Dynamic Scene Reconstruction](https://mlsanity.com/paper/2405.17891/) (2024)
- [Projecting Gaussian Ellipsoids While Avoiding Affine Projection Approximation](https://mlsanity.com/paper/2411.07579/) (2024)
- [ERUPT: Efficient Rendering with Unposed Patch Transformer](https://mlsanity.com/paper/2503.24374/) (2025)
- [Towards High-Fidelity Gaussian Splatting with Queried-Convolution Neural Networks](https://mlsanity.com/paper/2512.12898/) (2025)
- [QuantumGS: Quantum Encoding Framework for Gaussian Splatting](https://mlsanity.com/paper/2602.05047/) (2026)
- [FastPano3D: Feed-Forward Indoor Panoramic 3D Reconstruction from a Single Image](https://mlsanity.com/paper/2606.30352/) (2026)
- [AugSplat: Radiance Field-Informed Gaussian Splatting for Sparse-View Settings](https://mlsanity.com/paper/2606.31556/) (2026)
- [RouteBridge: Reliability-Routed Bidirectional Distillation Between Neural Radiance Fields and 3D Gaussian Splatting](https://mlsanity.com/paper/2609.09606/) (2026)
- [Is Gaussian Splatting Becoming Neural Again? A Taxonomy and Controlled Study of Learned Parameterization](https://mlsanity.com/paper/2609.12395/) (2026)
- [NRF-GS: Neural Residual Fields for Expressive and Compact Gaussian Splatting](https://mlsanity.com/paper/2609.37115/) (2026)

## Abstract (arXiv)

Neural Radiance Fields (NeRFs) have demonstrated the remarkable potential of neural networks to capture the intricacies of 3D objects. By encoding the shape and color information within neural network weights, NeRFs excel at producing strikingly sharp novel views of 3D objects. Recently, numerous generalizations of NeRFs utilizing generative models have emerged, expanding its versatility. In contrast, Gaussian Splatting (GS) offers a similar render quality with faster training and inference as it does not need neural networks to work. It encodes information about the 3D objects in the set of Gaussian distributions that can be rendered in 3D similarly to classical meshes. Unfortunately, GS are difficult to condition since they usually require circa hundred thousand Gaussian components. To mitigate the caveats of both models, we propose a hybrid model Viewing Direction Gaussian Splatting (VDGS) that uses GS representation of the 3D object's shape and NeRF-based encoding of color and opacity. Our model uses Gaussian distributions with trainable positions (i.e. means of Gaussian), shape (i.e. covariance of Gaussian), color and opacity, and a neural network that takes Gaussian parameters and viewing direction to produce changes in the said color and opacity. As a result, our model better describes shadows, light reflections, and the transparency of 3D objects without adding additional texture and light components.