# Learning Object-Centric Neural Scattering Functions for Free-Viewpoint Relighting and Scene Composition

arXiv: https://arxiv.org/abs/2303.06138 · HTML: https://mlsanity.com/paper/2303.06138/

## Key facts

- Cited by 17 papers in the map (data through 14 Oct 2026).
- Builds on 39 of its 76 references that are in the map.
- First posted on arXiv 10 Mar 2023.

## Builds on (most cited first, 30 of 39)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [The Unreasonable Effectiveness of Deep Features as a Perceptual Metric](https://mlsanity.com/paper/1801.03924/) (2018)
- [ShapeNet: An Information-Rich 3D Model Repository](https://mlsanity.com/paper/1512.03012/) (2015)
- [DeepSDF: Learning Continuous Signed Distance Functions for Shape Representation](https://mlsanity.com/paper/1901.05103/) (2019)
- [Occupancy Networks: Learning 3D Reconstruction in Function Space](https://mlsanity.com/paper/1812.03828/) (2018)
- [Mip-NeRF 360: Unbounded Anti-Aliased Neural Radiance Fields](https://mlsanity.com/paper/2111.12077/) (2021)
- [Mip-NeRF: A Multiscale Representation for Anti-Aliasing Neural Radiance Fields](https://mlsanity.com/paper/2103.13415/) (2021)
- [Neural Sparse Voxel Fields](https://mlsanity.com/paper/2007.11571/) (2020)
- [PlenOctrees for Real-time Rendering of Neural Radiance Fields](https://mlsanity.com/paper/2103.14024/) (2021)
- [Scene Representation Networks: Continuous 3D-Structure-Aware Neural Scene Representations](https://mlsanity.com/paper/1906.01618/) (2019)
- [NeRF++: Analyzing and Improving Neural Radiance Fields](https://mlsanity.com/paper/2010.07492/) (2020)
- [GIRAFFE: Representing Scenes as Compositional Generative Neural Feature Fields](https://mlsanity.com/paper/2011.12100/) (2020)
- [KiloNeRF: Speeding up Neural Radiance Fields with Thousands of Tiny MLPs](https://mlsanity.com/paper/2103.13744/) (2021)
- [UNISURF: Unifying Neural Implicit Surfaces and Radiance Fields for Multi-View Reconstruction](https://mlsanity.com/paper/2104.10078/) (2021)
- [Ref-NeRF: Structured View-Dependent Appearance for Neural Radiance Fields](https://mlsanity.com/paper/2112.03907/) (2021)
- [FastNeRF: High-Fidelity Neural Rendering at 200FPS](https://mlsanity.com/paper/2103.10380/) (2021)
- [NeRV: Neural Reflectance and Visibility Fields for Relighting and View Synthesis](https://mlsanity.com/paper/2012.03927/) (2020)
- [NeRD: Neural Reflectance Decomposition from Image Collections](https://mlsanity.com/paper/2012.03918/) (2020)
- [U2-Net: Going Deeper with Nested U-Structure for Salient Object Detection](https://mlsanity.com/paper/2005.09007/) (2020)
- [DeepVoxels: Learning Persistent 3D Feature Embeddings](https://mlsanity.com/paper/1812.01024/) (2018)
- [NeRF in the Dark: High Dynamic Range View Synthesis from Noisy Raw Images](https://mlsanity.com/paper/2111.13679/) (2021)
- [PhySG: Inverse Rendering with Spherical Gaussians for Physics-based Material Editing and Relighting](https://mlsanity.com/paper/2104.00674/) (2021)
- [Neural Point-Based Graphics](https://mlsanity.com/paper/1906.08240/) (2019)
- [Neural Scene Graphs for Dynamic Scenes](https://mlsanity.com/paper/2011.10379/) (2020)
- [State of the Art on Neural Rendering](https://mlsanity.com/paper/2004.03805/) (2020)
- [Learning Object-Compositional Neural Radiance Field for Editable Scene Rendering](https://mlsanity.com/paper/2109.01847/) (2021)
- [Neural Descriptor Fields: SE(3)-Equivariant Object Representations for Manipulation](https://mlsanity.com/paper/2112.05124/) (2021)
- [Neural-PIL: Neural Pre-Integrated Lighting for Reflectance Decomposition](https://mlsanity.com/paper/2110.14373/) (2021)
- [Neural Reflectance Fields for Appearance Acquisition](https://mlsanity.com/paper/2008.03824/) (2020)
- [Single Image Portrait Relighting](https://mlsanity.com/paper/1905.00824/) (2019)

## Built on by (most cited first, 17 of 17)

- [Stanford-ORB: A Real-World 3D Object Inverse Rendering Benchmark](https://mlsanity.com/paper/2310.16044/) (2023)
- [GS^3: Efficient Relighting with Triple Gaussian Splatting](https://mlsanity.com/paper/2410.11419/) (2024)
- [Digital Twin Catalog: A Large-Scale Photorealistic 3D Object Digital Twin Dataset](https://mlsanity.com/paper/2504.08541/) (2025)
- [Inferring Hybrid Neural Fluid Fields from Videos](https://mlsanity.com/paper/2312.06561/) (2023)
- [Fluid Simulation on Neural Flow Maps](https://mlsanity.com/paper/2312.14635/) (2023)
- [Multi-Object Manipulation via Object-Centric Neural Scattering Functions](https://mlsanity.com/paper/2306.08748/) (2023)
- [BiGS: Bidirectional Gaussian Primitives for Relightable 3D Gaussian Splatting](https://mlsanity.com/paper/2408.13370/) (2024)
- [Fast Image-based Neural Relighting with Translucency-Reflection Modeling](https://mlsanity.com/paper/2306.09322/) (2023)
- [Relighting Scenes with Object Insertions in Neural Radiance Fields](https://mlsanity.com/paper/2406.14806/) (2024)
- [Closing the Visual Sim-to-Real Gap with Object-Composable NeRFs](https://mlsanity.com/paper/2403.04114/) (2024)
- [Subsurface Scattering for 3D Gaussian Splatting](https://mlsanity.com/paper/2408.12282/) (2024)
- [Neural Acquisition & Representation of Subsurface Scattering](https://mlsanity.com/paper/2606.02292/) (2026)
- [Visual Tomography: Physically Faithful Volumetric Models of Partially Translucent Objects](https://mlsanity.com/paper/2312.13494/) (2023)
- [DIAMOND-SSS: Diffusion-Augmented Multi-View Optimization for Data-efficient SubSurface Scattering](https://mlsanity.com/paper/2601.12020/) (2026)
- [From Phase to Phenomenon: Self-Supervised Learning of Subsurface Scattering with Minimal Phase-shift Inputs](https://mlsanity.com/paper/2606.29461/) (2026)
- [Objects as Audio-Visual Modal Sound Fields](https://mlsanity.com/paper/2608.05145/) (2026)
- [VolS-GS: Relightable Gaussian Splatting with Volumetric Subsurface Scattering](https://mlsanity.com/paper/2610.04007/) (2026)

## Abstract (arXiv)

Photorealistic object appearance modeling from 2D images is a constant topic in vision and graphics. While neural implicit methods (such as Neural Radiance Fields) have shown high-fidelity view synthesis results, they cannot relight the captured objects. More recent neural inverse rendering approaches have enabled object relighting, but they represent surface properties as simple BRDFs, and therefore cannot handle translucent objects. We propose Object-Centric Neural Scattering Functions (OSFs) for learning to reconstruct object appearance from only images. OSFs not only support free-viewpoint object relighting, but also can model both opaque and translucent objects. While accurately modeling subsurface light transport for translucent objects can be highly complex and even intractable for neural methods, OSFs learn to approximate the radiance transfer from a distant light to an outgoing direction at any spatial location. This approximation avoids explicitly modeling complex subsurface scattering, making learning a neural implicit model tractable. Experiments on real and synthetic data show that OSFs accurately reconstruct appearances for both opaque and translucent objects, allowing faithful free-viewpoint relighting as well as scene composition.