# NeRF in detail: Learning to sample for view synthesis

arXiv: https://arxiv.org/abs/2106.05264 · HTML: https://mlsanity.com/paper/2106.05264/

## Key facts

- Cited by 35 papers in the map (data through 14 Oct 2026).
- Builds on 39 of its 71 references that are in the map.
- First posted on arXiv 9 Jun 2021.

## Builds on (most cited first, 30 of 39)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks](https://mlsanity.com/paper/1506.01497/) (2015)
- [End-to-End Object Detection with Transformers](https://mlsanity.com/paper/2005.12872/) (2020)
- [Fast R-CNN](https://mlsanity.com/paper/1504.08083/) (2015)
- [DeepSDF: Learning Continuous Signed Distance Functions for Shape Representation](https://mlsanity.com/paper/1901.05103/) (2019)
- [Rich feature hierarchies for accurate object detection and semantic segmentation](https://mlsanity.com/paper/1311.2524/) (2013)
- [Occupancy Networks: Learning 3D Reconstruction in Function Space](https://mlsanity.com/paper/1812.03828/) (2018)
- [Mip-NeRF: A Multiscale Representation for Anti-Aliasing Neural Radiance Fields](https://mlsanity.com/paper/2103.13415/) (2021)
- [pixelNeRF: Neural Radiance Fields from One or Few Images](https://mlsanity.com/paper/2012.02190/) (2020)
- [D-NeRF: Neural Radiance Fields for Dynamic Scenes](https://mlsanity.com/paper/2011.13961/) (2020)
- [MLP-Mixer: An all-MLP Architecture for Vision](https://mlsanity.com/paper/2105.01601/) (2021)
- [NeRF in the Wild: Neural Radiance Fields for Unconstrained Photo Collections](https://mlsanity.com/paper/2008.02268/) (2020)
- [Neural Sparse Voxel Fields](https://mlsanity.com/paper/2007.11571/) (2020)
- [PlenOctrees for Real-time Rendering of Neural Radiance Fields](https://mlsanity.com/paper/2103.14024/) (2021)
- [Scene Representation Networks: Continuous 3D-Structure-Aware Neural Scene Representations](https://mlsanity.com/paper/1906.01618/) (2019)
- [NeRF++: Analyzing and Improving Neural Radiance Fields](https://mlsanity.com/paper/2010.07492/) (2020)
- [Neural Scene Flow Fields for Space-Time View Synthesis of Dynamic Scenes](https://mlsanity.com/paper/2011.13084/) (2020)
- [pi-GAN: Periodic Implicit Generative Adversarial Networks for 3D-Aware Image Synthesis](https://mlsanity.com/paper/2012.00926/) (2020)
- [IBRNet: Learning Multi-View Image-Based Rendering](https://mlsanity.com/paper/2102.13090/) (2021)
- [Differentiable Volumetric Rendering: Learning Implicit 3D Representations without 3D Supervision](https://mlsanity.com/paper/1912.07372/) (2019)
- [GRAF: Generative Radiance Fields for 3D-Aware Image Synthesis](https://mlsanity.com/paper/2007.02442/) (2020)
- [KiloNeRF: Speeding up Neural Radiance Fields with Thousands of Tiny MLPs](https://mlsanity.com/paper/2103.13744/) (2021)
- [Local Light Field Fusion: Practical View Synthesis with Prescriptive Sampling Guidelines](https://mlsanity.com/paper/1905.00889/) (2019)
- [FastNeRF: High-Fidelity Neural Rendering at 200FPS](https://mlsanity.com/paper/2103.10380/) (2021)
- [Baking Neural Radiance Fields for Real-Time View Synthesis](https://mlsanity.com/paper/2103.14645/) (2021)
- [Space-time Neural Irradiance Fields for Free-Viewpoint Video](https://mlsanity.com/paper/2011.12950/) (2020)
- [Neural Point-Based Graphics](https://mlsanity.com/paper/1906.08240/) (2019)
- [Temporal Action Detection with Structured Segment Networks](https://mlsanity.com/paper/1704.06228/) (2017)
- [Free View Synthesis](https://mlsanity.com/paper/2008.05511/) (2020)

## Built on by (most cited first, 30 of 35)

- [Mip-NeRF 360: Unbounded Anti-Aliased Neural Radiance Fields](https://mlsanity.com/paper/2111.12077/) (2021)
- [Neural Fields in Visual Computing and Beyond](https://mlsanity.com/paper/2111.11426/) (2021)
- [Fast Dynamic Radiance Fields with Time-Aware Neural Voxels](https://mlsanity.com/paper/2205.15285/) (2022)
- [GeoNeRF: Generalizing NeRF with Geometry Priors](https://mlsanity.com/paper/2111.13539/) (2021)
- [Efficient Neural Radiance Fields for Interactive Free-viewpoint Video](https://mlsanity.com/paper/2112.01517/) (2021)
- [ActiveNeRF: Learning where to See with Uncertainty Estimation](https://mlsanity.com/paper/2209.08546/) (2022)
- [NeRF-Art: Text-Driven Neural Radiance Fields Stylization](https://mlsanity.com/paper/2212.08070/) (2022)
- [Vision Transformer for NeRF-Based View Synthesis from a Single Input Image](https://mlsanity.com/paper/2207.05736/) (2022)
- [TermiNeRF: Ray Termination Prediction for Efficient Neural Rendering](https://mlsanity.com/paper/2111.03643/) (2021)
- [Neural Point Light Fields](https://mlsanity.com/paper/2112.01473/) (2021)
- [Strivec: Sparse Tri-Vector Radiance Fields](https://mlsanity.com/paper/2307.13226/) (2023)
- [NeurMiPs: Neural Mixture of Planar Experts for View Synthesis](https://mlsanity.com/paper/2204.13696/) (2022)
- [NeuSample: Neural Sample Field for Efficient View Synthesis](https://mlsanity.com/paper/2111.15552/) (2021)
- [VRP-UDF: Towards Unbiased Learning of Unsigned Distance Functions from Multi-view Images with Volume Rendering Priors](https://mlsanity.com/paper/2407.16396/) (2024)
- [Nonparametric Teaching of Implicit Neural Representations](https://mlsanity.com/paper/2405.10531/) (2024)
- [Neural Pruning for 3D Scene Reconstruction: Efficient NeRF Acceleration](https://mlsanity.com/paper/2504.00950/) (2025)
- [Multi-View Reconstruction using Signed Ray Distance Functions (SRDF)](https://mlsanity.com/paper/2209.00082/) (2022)
- [Deep Learning for Accelerated and Robust MRI Reconstruction: a Review](https://mlsanity.com/paper/2404.15692/) (2024)
- [Neural Radiance Fields: Past, Present, and Future](https://mlsanity.com/paper/2304.10050/) (2023)
- [Efficient Ray Sampling for Radiance Fields Reconstruction](https://mlsanity.com/paper/2308.15547/) (2023)
- [ABLE-NeRF: Attention-Based Rendering with Learnable Embeddings for Neural Radiance Field](https://mlsanity.com/paper/2303.13817/) (2023)
- [SO-NeRF: Active View Planning for NeRF using Surrogate Objectives](https://mlsanity.com/paper/2312.03266/) (2023)
- [STDR: Spatio-Temporal Decoupling for Real-Time Dynamic Scene Rendering](https://mlsanity.com/paper/2505.22400/) (2025)
- [L0-Sampler: An L0 Model Guided Volume Sampling for NeRF](https://mlsanity.com/paper/2311.07044/) (2023)
- [Differentiable Rendering with Reparameterized Volume Sampling](https://mlsanity.com/paper/2302.10970/) (2023)
- [Analyzing the Internals of Neural Radiance Fields](https://mlsanity.com/paper/2306.00696/) (2023)
- [TSDF-Sampling: Efficient Sampling for Neural Surface Field using Truncated Signed Distance Field](https://mlsanity.com/paper/2311.17878/) (2023)
- [GL-NeRF: Gauss-Laguerre Quadrature Enables Training-Free NeRF Acceleration](https://mlsanity.com/paper/2410.19831/) (2024)
- [NERFIFY: A Multi-Agent Framework for Turning NeRF Papers into Code](https://mlsanity.com/paper/2603.00805/) (2026)
- [LoLep: Single-View View Synthesis with Locally-Learned Planes and Self-Attention Occlusion Inference](https://mlsanity.com/paper/2307.12217/) (2023)

## Abstract (arXiv)

Neural radiance fields (NeRF) methods have demonstrated impressive novel view synthesis performance. The core approach is to render individual rays by querying a neural network at points sampled along the ray to obtain the density and colour of the sampled points, and integrating this information using the rendering equation. Since dense sampling is computationally prohibitive, a common solution is to perform coarse-to-fine sampling. In this work we address a clear limitation of the vanilla coarse-to-fine approach -- that it is based on a heuristic and not trained end-to-end for the task at hand. We introduce a differentiable module that learns to propose samples and their importance for the fine network, and consider and compare multiple alternatives for its neural architecture. Training the proposal module from scratch can be unstable due to lack of supervision, so an effective pre-training strategy is also put forward. The approach, named `NeRF in detail' (NeRF-ID), achieves superior view synthesis quality over NeRF and the state-of-the-art on the synthetic Blender benchmark and on par or better performance on the real LLFF-NeRF scenes. Furthermore, by leveraging the predicted sample importance, a 25% saving in computation can be achieved without significantly sacrificing the rendering quality.