# Neural Projection Mapping Using Reflectance Fields

arXiv: https://arxiv.org/abs/2306.06595 · HTML: https://mlsanity.com/paper/2306.06595/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- Builds on 13 of its 52 references that are in the map.
- First posted on arXiv 11 Jun 2023.

## Builds on (most cited first, 13 of 13)

- [Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2006.11239/) (2020)
- [High-Resolution Image Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2112.10752/) (2021)
- [DreamFusion: Text-to-3D using 2D Diffusion](https://mlsanity.com/paper/2209.14988/) (2022)
- [Ref-NeRF: Structured View-Dependent Appearance for Neural Radiance Fields](https://mlsanity.com/paper/2112.03907/) (2021)
- [NeRV: Neural Reflectance and Visibility Fields for Relighting and View Synthesis](https://mlsanity.com/paper/2012.03927/) (2020)
- [U2-Net: Going Deeper with Nested U-Structure for Salient Object Detection](https://mlsanity.com/paper/2005.09007/) (2020)
- [CLIP-NeRF: Text-and-Image Driven Manipulation of Neural Radiance Fields](https://mlsanity.com/paper/2112.05139/) (2021)
- [Extracting Triangular 3D Models, Materials, and Lighting From Images](https://mlsanity.com/paper/2111.12503/) (2021)
- [Neural Reflectance Fields for Appearance Acquisition](https://mlsanity.com/paper/2008.03824/) (2020)
- [NerfAcc: A General NeRF Acceleration Toolbox](https://mlsanity.com/paper/2210.04847/) (2022)
- [PS-NeRF: Neural Inverse Rendering for Multi-view Photometric Stereo](https://mlsanity.com/paper/2207.11406/) (2022)
- [Cross-domain Compositing with Pretrained Diffusion Models](https://mlsanity.com/paper/2302.10167/) (2023)
- [End-to-end Full Projector Compensation](https://mlsanity.com/paper/2008.00965/) (2020)

## Built on by (most cited first, 8 of 8)

- [GS-ProCams: Gaussian Splatting-based Projector-Camera Systems](https://mlsanity.com/paper/2412.11762/) (2024)
- [DPCS: Path Tracing-Based Differentiable Projector-Camera Systems](https://mlsanity.com/paper/2503.12174/) (2025)
- [LAPIG: Language Guided Projector Image Generation with Surface Adaptation and Stylization](https://mlsanity.com/paper/2503.12173/) (2025)
- [ProCap: Projection-Aware Captioning for Spatial Augmented Reality](https://mlsanity.com/paper/2604.00912/) (2026)
- [Setup-Independent Full Projector Compensation](https://mlsanity.com/paper/2604.01736/) (2026)
- [Breaking the Scalability Limit of Multi-Projector Calibration with Embedded Cameras](https://mlsanity.com/paper/2604.24024/) (2026)
- [Physically-guided Image Generation for Multi-Projection Mapping](https://mlsanity.com/paper/2606.22477/) (2026)
- [FF-ProCams: Feed-Forward Gaussian Splatting for Projector-Camera System](https://mlsanity.com/paper/2607.17803/) (2026)

## Abstract (arXiv)

We introduce a high resolution spatially adaptive light source, or a projector, into a neural reflectance field that allows to both calibrate the projector and photo realistic light editing. The projected texture is fully differentiable with respect to all scene parameters, and can be optimized to yield a desired appearance suitable for applications in augmented reality and projection mapping. Our neural field consists of three neural networks, estimating geometry, material, and transmittance. Using an analytical BRDF model and carefully selected projection patterns, our acquisition process is simple and intuitive, featuring a fixed uncalibrated projected and a handheld camera with a co-located light source. As we demonstrate, the virtual projector incorporated into the pipeline improves scene understanding and enables various projection mapping applications, alleviating the need for time consuming calibration steps performed in a traditional setting per view or projector location. In addition to enabling novel viewpoint synthesis, we demonstrate state-of-the-art performance projector compensation for novel viewpoints, improvement over the baselines in material and scene reconstruction, and three simply implemented scenarios where projection image optimization is performed, including the use of a 2D generative model to consistently dictate scene appearance from multiple viewpoints. We believe that neural projection mapping opens up the door to novel and exciting downstream tasks, through the joint optimization of the scene and projection images.