# Objects With Lighting: A Real-World Dataset for Evaluating Reconstruction and Rendering for Object Relighting

arXiv: https://arxiv.org/abs/2401.09126 · HTML: https://mlsanity.com/paper/2401.09126/

## Key facts

- Cited by 18 papers in the map (data through 14 Oct 2026).
- Builds on 36 of its 79 references that are in the map.
- First posted on arXiv 17 Jan 2024.

## Builds on (most cited first, 30 of 36)

- [The Unreasonable Effectiveness of Deep Features as a Perceptual Metric](https://mlsanity.com/paper/1801.03924/) (2018)
- [NeuS: Learning Neural Implicit Surfaces by Volume Rendering for Multi-view Reconstruction](https://mlsanity.com/paper/2106.10689/) (2021)
- [Plenoxels: Radiance Fields without Neural Networks](https://mlsanity.com/paper/2112.05131/) (2021)
- [PlenOctrees for Real-time Rendering of Neural Radiance Fields](https://mlsanity.com/paper/2103.14024/) (2021)
- [The Replica Dataset: A Digital Replica of Indoor Spaces](https://mlsanity.com/paper/1906.05797/) (2019)
- [Hypersim: A Photorealistic Synthetic Dataset for Holistic Indoor Scene Understanding](https://mlsanity.com/paper/2011.02523/) (2020)
- [NeRV: Neural Reflectance and Visibility Fields for Relighting and View Synthesis](https://mlsanity.com/paper/2012.03927/) (2020)
- [BlendedMVS: A Large-scale Dataset for Generalized Multi-view Stereo Networks](https://mlsanity.com/paper/1911.10127/) (2019)
- [NeRD: Neural Reflectance Decomposition from Image Collections](https://mlsanity.com/paper/2012.03918/) (2020)
- [Extracting Triangular 3D Models, Materials, and Lighting From Images](https://mlsanity.com/paper/2111.12503/) (2021)
- [PhySG: Inverse Rendering with Spherical Gaussians for Physics-based Material Editing and Relighting](https://mlsanity.com/paper/2104.00674/) (2021)
- [NeX: Real-time View Synthesis with Neural Basis Expansion](https://mlsanity.com/paper/2103.05606/) (2021)
- [Neural-PIL: Neural Pre-Integrated Lighting for Reflectance Decomposition](https://mlsanity.com/paper/2110.14373/) (2021)
- [Neural Reflectance Fields for Appearance Acquisition](https://mlsanity.com/paper/2008.03824/) (2020)
- [Modeling Indirect Illumination for Inverse Rendering](https://mlsanity.com/paper/2204.06837/) (2022)
- [TensoIR: Tensorial Inverse Rendering](https://mlsanity.com/paper/2304.12461/) (2023)
- [Shape, Light, and Material Decomposition from Images using Monte Carlo Rendering and Denoising](https://mlsanity.com/paper/2206.03380/) (2022)
- [NeRO: Neural Geometry and BRDF Reconstruction of Reflective Objects from Multiview Images](https://mlsanity.com/paper/2305.17398/) (2023)
- [Single-Image SVBRDF Capture with a Rendering-Aware Deep Network](https://mlsanity.com/paper/1810.09718/) (2018)
- [Deep Reflectance Volumes: Relightable Reconstructions from Multi-View Photometric Images](https://mlsanity.com/paper/2007.09892/) (2020)
- [InteriorNet: Mega-scale Multi-sensor Photo-realistic Indoor Scenes Dataset](https://mlsanity.com/paper/1809.00716/) (2018)
- [GeoSim: Realistic Video Simulation via Geometry-Aware Composition for Self-Driving](https://mlsanity.com/paper/2101.06543/) (2021)
- [Rendering Synthetic Objects into Legacy Photographs](https://mlsanity.com/paper/1912.11565/) (2019)
- [Materials for Masses: SVBRDF Acquisition with a Single Mobile Phone Image](https://mlsanity.com/paper/1804.05790/) (2018)
- [Deep 3D Capture: Geometry and Reflectance from Sparse Multi-View Images](https://mlsanity.com/paper/2003.12642/) (2020)
- [Multi-View Photometric Stereo: A Robust Solution and Benchmark Dataset for Spatially Varying Isotropic Materials](https://mlsanity.com/paper/2001.06659/) (2020)
- [A Dataset of Multi-Illumination Images in the Wild](https://mlsanity.com/paper/1910.08131/) (2019)
- [ReLight My NeRF: A Dataset for Novel View Synthesis and Relighting of Real World Objects](https://mlsanity.com/paper/2304.10448/) (2023)
- [Generative Modelling of BRDF Textures from Flash Images](https://mlsanity.com/paper/2102.11861/) (2021)
- [Uncalibrated Neural Inverse Rendering for Photometric Stereo of General Surfaces](https://mlsanity.com/paper/2012.06777/) (2020)

## Built on by (most cited first, 18 of 18)

- [RelitLRM: Generative Relightable Radiance for Large Reconstruction Models](https://mlsanity.com/paper/2410.06231/) (2024)
- [LIRM: Large Inverse Rendering Model for Progressive Reconstruction of Shape, Materials and View-dependent Radiance Fields](https://mlsanity.com/paper/2504.20026/) (2025)
- [LightSwitch: Multi-view Relighting with Material-guided Diffusion](https://mlsanity.com/paper/2508.06494/) (2025)
- [LuxDiT: Lighting Estimation with Video Diffusion Transformer](https://mlsanity.com/paper/2509.03680/) (2025)
- [GUS-IR: Gaussian Splatting with Unified Shading for Inverse Rendering](https://mlsanity.com/paper/2411.07478/) (2024)
- [Generative World Renderer](https://mlsanity.com/paper/2604.02329/) (2026)
- [MV-CoLight: Efficient Object Compositing with Consistent Lighting and Shadow Generation](https://mlsanity.com/paper/2505.21483/) (2025)
- [Inverse Rendering using Multi-Bounce Path Tracing and Reservoir Sampling](https://mlsanity.com/paper/2406.16360/) (2024)
- [LSRM: High-Fidelity Object-Centric Reconstruction via Scaled Context Windows](https://mlsanity.com/paper/2604.05182/) (2026)
- [Material Transforms from Disentangled NeRF Representations](https://mlsanity.com/paper/2411.08037/) (2024)
- [UAVLight: A Benchmark for Illumination-Robust 3D Reconstruction in Unmanned Aerial Vehicle (UAV) Scenes](https://mlsanity.com/paper/2511.21565/) (2025)
- [Learning Latent Proxies for Controllable Single-Image Relighting](https://mlsanity.com/paper/2603.15555/) (2026)
- [F-RNG: Feed-Forward Relightable Neural Gaussians](https://mlsanity.com/paper/2605.25975/) (2026)
- [DreamLifting: A Plug-in Module Lifting MV Diffusion Models for 3D Asset Generation](https://mlsanity.com/paper/2509.07435/) (2025)
- [Spec-Gloss Surfels and Normal-Diffuse Priors for Relightable Glossy Objects](https://mlsanity.com/paper/2510.02069/) (2025)
- [EvalMVX: A Unified Benchmarking for Neural 3D Reconstruction under Diverse Multiview Setups](https://mlsanity.com/paper/2602.24065/) (2026)
- [WildRelight: A Real-World Benchmark and Physics-Guided Adaptation for Single-Image Relighting](https://mlsanity.com/paper/2605.11696/) (2026)
- [Ambient-robust Inverse Rendering using Active RGB-NIR Imaging](https://mlsanity.com/paper/2605.30250/) (2026)

## Abstract (arXiv)

Reconstructing an object from photos and placing it virtually in a new environment goes beyond the standard novel view synthesis task as the appearance of the object has to not only adapt to the novel viewpoint but also to the new lighting conditions and yet evaluations of inverse rendering methods rely on novel view synthesis data or simplistic synthetic datasets for quantitative analysis. This work presents a real-world dataset for measuring the reconstruction and rendering of objects for relighting. To this end, we capture the environment lighting and ground truth images of the same objects in multiple environments allowing to reconstruct the objects from images taken in one environment and quantify the quality of the rendered views for the unseen lighting environments. Further, we introduce a simple baseline composed of off-the-shelf methods and test several state-of-the-art methods on the relighting task and show that novel view synthesis is not a reliable proxy to measure performance. Code and dataset are available at https://github.com/isl-org/objects-with-lighting .