# Differentiable Point-Based Radiance Fields for Efficient View Synthesis

arXiv: https://arxiv.org/abs/2205.14330 · HTML: https://mlsanity.com/paper/2205.14330/

## Key facts

- Cited by 86 papers in the map (data through 14 Oct 2026).
- Builds on 27 of its 48 references that are in the map.
- First posted on arXiv 28 May 2022.

## Builds on (most cited first, 27 of 27)

- [Plenoxels: Radiance Fields without Neural Networks](https://mlsanity.com/paper/2112.05131/) (2021)
- [pixelNeRF: Neural Radiance Fields from One or Few Images](https://mlsanity.com/paper/2012.02190/) (2020)
- [PlenOctrees for Real-time Rendering of Neural Radiance Fields](https://mlsanity.com/paper/2103.14024/) (2021)
- [MVSNet: Depth Inference for Unstructured Multi-view Stereo](https://mlsanity.com/paper/1804.02505/) (2018)
- [MVSNeRF: Fast Generalizable Radiance Field Reconstruction from Multi-View Stereo](https://mlsanity.com/paper/2103.15595/) (2021)
- [IBRNet: Learning Multi-View Image-Based Rendering](https://mlsanity.com/paper/2102.13090/) (2021)
- [KiloNeRF: Speeding up Neural Radiance Fields with Thousands of Tiny MLPs](https://mlsanity.com/paper/2103.13744/) (2021)
- [FastNeRF: High-Fidelity Neural Rendering at 200FPS](https://mlsanity.com/paper/2103.10380/) (2021)
- [Point-NeRF: Point-based Neural Radiance Fields](https://mlsanity.com/paper/2201.08845/) (2022)
- [Baking Neural Radiance Fields for Real-Time View Synthesis](https://mlsanity.com/paper/2103.14645/) (2021)
- [SynSin: End-to-end View Synthesis from a Single Image](https://mlsanity.com/paper/1912.08804/) (2019)
- [Neural Point-Based Graphics](https://mlsanity.com/paper/1906.08240/) (2019)
- [Mixture of Volumetric Primitives for Efficient Neural Rendering](https://mlsanity.com/paper/2103.01954/) (2021)
- [Differentiable Surface Splatting for Point-based Geometry Processing](https://mlsanity.com/paper/1906.04173/) (2019)
- [DONeRF: Towards Real-Time Rendering of Compact Neural Radiance Fields using Depth Oracle Networks](https://mlsanity.com/paper/2103.03231/) (2021)
- [NeX: Real-time View Synthesis with Neural Basis Expansion](https://mlsanity.com/paper/2103.05606/) (2021)
- [Stereo Magnification: Learning View Synthesis using Multiplane Images](https://mlsanity.com/paper/1805.09817/) (2018)
- [Neural Rays for Occlusion-aware Image-based Rendering](https://mlsanity.com/paper/2107.13421/) (2021)
- [Point-Based Neural Rendering with Per-View Optimization](https://mlsanity.com/paper/2109.02369/) (2021)
- [Editable Free-viewpoint Video Using a Layered Neural Representation](https://mlsanity.com/paper/2104.14786/) (2021)
- [Neural Rerendering in the Wild](https://mlsanity.com/paper/1904.04290/) (2019)
- [DPSNet: End-to-end Deep Plane Sweep Stereo](https://mlsanity.com/paper/1905.00538/) (2019)
- [DRWR: A Differentiable Renderer without Rendering for Unsupervised 3D Structure Learning from Silhouette Images](https://mlsanity.com/paper/2007.06127/) (2020)
- [Neural BRDF Representation and Importance Sampling](https://mlsanity.com/paper/2102.05963/) (2021)
- [Invertible Neural BRDF for Object Inverse Rendering](https://mlsanity.com/paper/2008.04030/) (2020)
- [VaxNeRF: Revisiting the Classic for Voxel-Accelerated Neural Radiance Field](https://mlsanity.com/paper/2111.13112/) (2021)
- [ADOP: Approximate Differentiable One-Pixel Point Rendering](https://mlsanity.com/paper/2110.06635/) (2021)

## Built on by (most cited first, 30 of 86)

- [3D Gaussian Splatting for Real-Time Radiance Field Rendering](https://mlsanity.com/paper/2308.04079/) (2023)
- [Deformable 3D Gaussians for High-Fidelity Monocular Dynamic Scene Reconstruction](https://mlsanity.com/paper/2309.13101/) (2023)
- [Dynamic 3D Gaussians: Tracking by Persistent Dynamic View Synthesis](https://mlsanity.com/paper/2308.09713/) (2023)
- [Compact 3D Gaussian Representation for Radiance Field](https://mlsanity.com/paper/2311.13681/) (2023)
- [SC-GS: Sparse-Controlled Gaussian Splatting for Editable Dynamic Scenes](https://mlsanity.com/paper/2312.14937/) (2023)
- [Street Gaussians: Modeling Dynamic Urban Scenes with Gaussian Splatting](https://mlsanity.com/paper/2401.01339/) (2024)
- [Gaussian-Flow: 4D Reconstruction with Dynamic 3D Gaussian Particle](https://mlsanity.com/paper/2312.03431/) (2023)
- [3DGS-Avatar: Animatable Avatars via Deformable 3D Gaussian Splatting](https://mlsanity.com/paper/2312.09228/) (2023)
- [COLMAP-Free 3D Gaussian Splatting](https://mlsanity.com/paper/2312.07504/) (2023)
- [4D-Rotor Gaussian Splatting: Towards Efficient Novel View Synthesis for Dynamic Scenes](https://mlsanity.com/paper/2402.03307/) (2024)
- [LVSM: A Large View Synthesis Model with Minimal 3D Inductive Bias](https://mlsanity.com/paper/2410.17242/) (2024)
- [PointAvatar: Deformable Point-based Head Avatars from Videos](https://mlsanity.com/paper/2212.08377/) (2022)
- [GauHuman: Articulated Gaussian Splatting from Monocular Human Videos](https://mlsanity.com/paper/2312.02973/) (2023)
- [Relightable 3D Gaussians: Realistic Point Cloud Relighting with BRDF Decomposition and Ray Tracing](https://mlsanity.com/paper/2311.16043/) (2023)
- [DynMF: Neural Motion Factorization for Real-time Dynamic View Synthesis with 3D Gaussian Splatting](https://mlsanity.com/paper/2312.00112/) (2023)
- [GART: Gaussian Articulated Template Models](https://mlsanity.com/paper/2311.16099/) (2023)
- [ContextGS: Compact 3D Gaussian Splatting with Anchor Level Context Model](https://mlsanity.com/paper/2405.20721/) (2024)
- [GauFRe: Gaussian Deformation Fields for Real-time Dynamic Novel View Synthesis](https://mlsanity.com/paper/2312.11458/) (2023)
- [GES: Generalized Exponential Splatting for Efficient Radiance Field Rendering](https://mlsanity.com/paper/2402.10128/) (2024)
- [4K4D: Real-Time 4D View Synthesis at 4K Resolution](https://mlsanity.com/paper/2310.11448/) (2023)
- [3DGSR: Implicit Surface Reconstruction with 3D Gaussian Splatting](https://mlsanity.com/paper/2404.00409/) (2024)
- [Spec-Gaussian: Anisotropic View-Dependent Appearance for 3D Gaussian Splatting](https://mlsanity.com/paper/2402.15870/) (2024)
- [SMERF: Streamable Memory Efficient Radiance Fields for Real-Time Large-Scene Exploration](https://mlsanity.com/paper/2312.07541/) (2023)
- [Gaussian Shell Maps for Efficient 3D Human Generation](https://mlsanity.com/paper/2311.17857/) (2023)
- [Neural Parametric Gaussians for Monocular Non-Rigid Object Reconstruction](https://mlsanity.com/paper/2312.01196/) (2023)
- [Masked Wavelet Representation for Compact Neural Radiance Fields](https://mlsanity.com/paper/2212.09069/) (2022)
- [Gaussian Splatting: 3D Reconstruction and Novel View Synthesis, a Review](https://mlsanity.com/paper/2405.03417/) (2024)
- [Binary Radiance Fields](https://mlsanity.com/paper/2306.07581/) (2023)
- [ClimateNeRF: Extreme Weather Synthesis in Neural Radiance Field](https://mlsanity.com/paper/2211.13226/) (2022)
- [Dynamic Point Fields](https://mlsanity.com/paper/2304.02626/) (2023)

## Abstract (arXiv)

We propose a differentiable rendering algorithm for efficient novel view synthesis. By departing from volume-based representations in favor of a learned point representation, we improve on existing methods more than an order of magnitude in memory and runtime, both in training and inference. The method begins with a uniformly-sampled random point cloud and learns per-point position and view-dependent appearance, using a differentiable splat-based renderer to evolve the model to match a set of input images. Our method is up to 300x faster than NeRF in both training and inference, with only a marginal sacrifice in quality, while using less than 10~MB of memory for a static scene. For dynamic scenes, our method trains two orders of magnitude faster than STNeRF and renders at near interactive rate, while maintaining high image quality and temporal coherence even without imposing any temporal-coherency regularizers.