# Relighting Scenes with Object Insertions in Neural Radiance Fields

arXiv: https://arxiv.org/abs/2406.14806 · HTML: https://mlsanity.com/paper/2406.14806/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Builds on 47 of its 68 references that are in the map.
- First posted on arXiv 21 Jun 2024.

## Builds on (most cited first, 30 of 47)

- [The Unreasonable Effectiveness of Deep Features as a Perceptual Metric](https://mlsanity.com/paper/1801.03924/) (2018)
- [Mip-NeRF 360: Unbounded Anti-Aliased Neural Radiance Fields](https://mlsanity.com/paper/2111.12077/) (2021)
- [Mip-NeRF: A Multiscale Representation for Anti-Aliasing Neural Radiance Fields](https://mlsanity.com/paper/2103.13415/) (2021)
- [Plenoxels: Radiance Fields without Neural Networks](https://mlsanity.com/paper/2112.05131/) (2021)
- [D-NeRF: Neural Radiance Fields for Dynamic Scenes](https://mlsanity.com/paper/2011.13961/) (2020)
- [Direct Voxel Grid Optimization: Super-fast Convergence for Radiance Fields Reconstruction](https://mlsanity.com/paper/2111.11215/) (2021)
- [Neural Sparse Voxel Fields](https://mlsanity.com/paper/2007.11571/) (2020)
- [GIRAFFE: Representing Scenes as Compositional Generative Neural Feature Fields](https://mlsanity.com/paper/2011.12100/) (2020)
- [GRAF: Generative Radiance Fields for 3D-Aware Image Synthesis](https://mlsanity.com/paper/2007.02442/) (2020)
- [KiloNeRF: Speeding up Neural Radiance Fields with Thousands of Tiny MLPs](https://mlsanity.com/paper/2103.13744/) (2021)
- [Ref-NeRF: Structured View-Dependent Appearance for Neural Radiance Fields](https://mlsanity.com/paper/2112.03907/) (2021)
- [NeRV: Neural Reflectance and Visibility Fields for Relighting and View Synthesis](https://mlsanity.com/paper/2012.03927/) (2020)
- [StyleNeRF: A Style-based 3D-Aware Generator for High-resolution Image Synthesis](https://mlsanity.com/paper/2110.08985/) (2021)
- [NeRD: Neural Reflectance Decomposition from Image Collections](https://mlsanity.com/paper/2012.03918/) (2020)
- [INeRF: Inverting Neural Radiance Fields for Pose Estimation](https://mlsanity.com/paper/2012.05877/) (2020)
- [Decomposing NeRF for Editing via Feature Field Distillation](https://mlsanity.com/paper/2205.15585/) (2022)
- [SynSin: End-to-end View Synthesis from a Single Image](https://mlsanity.com/paper/1912.08804/) (2019)
- [PhySG: Inverse Rendering with Spherical Gaussians for Physics-based Material Editing and Relighting](https://mlsanity.com/paper/2104.00674/) (2021)
- [Learning Object-Compositional Neural Radiance Field for Editable Scene Rendering](https://mlsanity.com/paper/2109.01847/) (2021)
- [Editing Conditional Radiance Fields](https://mlsanity.com/paper/2105.06466/) (2021)
- [3D Photography using Context-aware Layered Depth Inpainting](https://mlsanity.com/paper/2004.04727/) (2020)
- [Neural-PIL: Neural Pre-Integrated Lighting for Reflectance Decomposition](https://mlsanity.com/paper/2110.14373/) (2021)
- [Neural Reflectance Fields for Appearance Acquisition](https://mlsanity.com/paper/2008.03824/) (2020)
- [Modeling Indirect Illumination for Inverse Rendering](https://mlsanity.com/paper/2204.06837/) (2022)
- [NeILF: Neural Incident Light Field for Physically-based Material Estimation](https://mlsanity.com/paper/2203.07182/) (2022)
- [InverseRenderNet: Learning single image inverse rendering](https://mlsanity.com/paper/1811.12328/) (2018)
- [Compressible-composable NeRF via Rank-residual Decomposition](https://mlsanity.com/paper/2205.14870/) (2022)
- [Object-Centric Neural Scene Rendering](https://mlsanity.com/paper/2012.08503/) (2020)
- [Neural Fields meet Explicit Geometric Representation for Inverse Rendering of Urban Scenes](https://mlsanity.com/paper/2304.03266/) (2023)
- [SAMURAI: Shape And Material from Unconstrained Real-world Arbitrary Image collections](https://mlsanity.com/paper/2205.15768/) (2022)

## Built on by (most cited first, 5 of 5)

- [CAD-NeRF: Learning NeRFs from Uncalibrated Few-view Images by CAD Model Retrieval](https://mlsanity.com/paper/2411.02979/) (2024)
- [Enhancing Novel View Synthesis from extremely sparse views with SfM-free 3D Gaussian Splatting Framework](https://mlsanity.com/paper/2508.15457/) (2025)
- [Lighting-Consistent Object Transfer Across Radiance Fields](https://mlsanity.com/paper/2606.22481/) (2026)
- [DecoupleGS: Interactive 3D Gaussian Splatting for End-to-End Autonomous Driving Testing](https://mlsanity.com/paper/2608.01761/) (2026)
- [FlashNormal: Detailed Surface Normal Estimation from Flash and No-Flash Images](https://mlsanity.com/paper/2608.25360/) (2026)

## Abstract (arXiv)

The insertion of objects into a scene and relighting are commonly utilized applications in augmented reality (AR). Previous methods focused on inserting virtual objects using CAD models or real objects from single-view images, resulting in highly limited AR application scenarios. We propose a novel NeRF-based pipeline for inserting object NeRFs into scene NeRFs, enabling novel view synthesis and realistic relighting, supporting physical interactions like casting shadows onto each other, from two sets of images depicting the object and scene. The lighting environment is in a hybrid representation of Spherical Harmonics and Spherical Gaussians, representing both high- and low-frequency lighting components very well, and supporting non-Lambertian surfaces. Specifically, we leverage the benefits of volume rendering and introduce an innovative approach for efficient shadow rendering by comparing the depth maps between the camera view and the light source view and generating vivid soft shadows. The proposed method achieves realistic relighting effects in extensive experimental evaluations.