# DPCS: Path Tracing-Based Differentiable Projector-Camera Systems

arXiv: https://arxiv.org/abs/2503.12174 · HTML: https://mlsanity.com/paper/2503.12174/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 11 of its 62 references that are in the map.
- First posted on arXiv 15 Mar 2025.

## Builds on (most cited first, 11 of 11)

- [The Unreasonable Effectiveness of Deep Features as a Perceptual Metric](https://mlsanity.com/paper/1801.03924/) (2018)
- [NeRV: Neural Reflectance and Visibility Fields for Relighting and View Synthesis](https://mlsanity.com/paper/2012.03927/) (2020)
- [Shape, Light, and Material Decomposition from Images using Monte Carlo Rendering and Denoising](https://mlsanity.com/paper/2206.03380/) (2022)
- [Inverse Path Tracing for Joint Material and Lighting Estimation](https://mlsanity.com/paper/1903.07145/) (2019)
- [Neural-PBIR Reconstruction of Shape, Material, and Illumination](https://mlsanity.com/paper/2304.13445/) (2023)
- [Physics-Based Inverse Rendering using Combined Implicit and Explicit Geometries](https://mlsanity.com/paper/2205.01242/) (2022)
- [End-to-end Full Projector Compensation](https://mlsanity.com/paper/2008.00965/) (2020)
- [Neural Projection Mapping Using Reflectance Fields](https://mlsanity.com/paper/2306.06595/) (2023)
- [CompenNet++: End-to-end Full Projector Compensation](https://mlsanity.com/paper/1908.06246/) (2019)
- [CompenHR: Efficient Full Compensation for High-resolution Projector](https://mlsanity.com/paper/2311.13409/) (2023)
- [Projection Mapping under Environmental Lighting by Replacing Room Lights with Heterogeneous Projectors](https://mlsanity.com/paper/2403.02547/) (2024)

## Built on by (most cited first, 4 of 4)

- [HumanMaterial: Human Material Estimation from a Single Image via Progressive Training](https://mlsanity.com/paper/2507.18385/) (2025)
- [ProCap: Projection-Aware Captioning for Spatial Augmented Reality](https://mlsanity.com/paper/2604.00912/) (2026)
- [Setup-Independent Full Projector Compensation](https://mlsanity.com/paper/2604.01736/) (2026)
- [FF-ProCams: Feed-Forward Gaussian Splatting for Projector-Camera System](https://mlsanity.com/paper/2607.17803/) (2026)

## Abstract (arXiv)

Projector-camera systems (ProCams) simulation aims to model the physical project-and-capture process and associated scene parameters of a ProCams, and is crucial for spatial augmented reality (SAR) applications such as ProCams relighting and projector compensation. Recent advances use an end-to-end neural network to learn the project-and-capture process. However, these neural network-based methods often implicitly encapsulate scene parameters, such as surface material, gamma, and white balance in the network parameters, and are less interpretable and hard for novel scene simulation. Moreover, neural networks usually learn the indirect illumination implicitly in an image-to-image translation way which leads to poor performance in simulating complex projection effects such as soft-shadow and interreflection. In this paper, we introduce a novel path tracing-based differentiable projector-camera systems (DPCS), offering a differentiable ProCams simulation method that explicitly integrates multi-bounce path tracing. Our DPCS models the physical project-and-capture process using differentiable physically-based rendering (PBR), enabling the scene parameters to be explicitly decoupled and learned using much fewer samples. Moreover, our physically-based method not only enables high-quality downstream ProCams tasks, such as ProCams relighting and projector compensation, but also allows novel scene simulation using the learned scene parameters. In experiments, DPCS demonstrates clear advantages over previous approaches in ProCams simulation, offering better interpretability, more efficient handling of complex interreflection and shadow, and requiring fewer training samples.