# People as Scene Probes

arXiv: https://arxiv.org/abs/2007.09209 · HTML: https://mlsanity.com/paper/2007.09209/

## Key facts

- Cited by 14 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 17 Jul 2020.

## Built on by (most cited first, 14 of 14)

- [GeoSim: Realistic Video Simulation via Geometry-Aware Composition for Self-Driving](https://mlsanity.com/paper/2101.06543/) (2021)
- [OBJECT 3DIT: Language-guided 3D-aware Image Editing](https://mlsanity.com/paper/2307.11073/) (2023)
- [Omnimatte: Associating Objects and Their Effects in Video](https://mlsanity.com/paper/2105.06993/) (2021)
- [Learning from Synthetic Shadows for Shadow Detection and Removal](https://mlsanity.com/paper/2101.01713/) (2021)
- [Controllable Shadow Generation Using Pixel Height Maps](https://mlsanity.com/paper/2207.05385/) (2022)
- [PixHt-Lab: Pixel Height Based Light Effect Generation for Image Compositing](https://mlsanity.com/paper/2303.00137/) (2023)
- [Lasagna: Layered Score Distillation for Disentangled Object Relighting](https://mlsanity.com/paper/2312.00833/) (2023)
- [Repopulating Street Scenes](https://mlsanity.com/paper/2103.16183/) (2021)
- [Temporal Feature Warping for Video Shadow Detection](https://mlsanity.com/paper/2107.14287/) (2021)
- [SAC-GAN: Structure-Aware Image Composition](https://mlsanity.com/paper/2112.06596/) (2021)
- [Animating Street View](https://mlsanity.com/paper/2310.08534/) (2023)
- [SIRfyN: Single Image Relighting from your Neighbors](https://mlsanity.com/paper/2112.04497/) (2021)
- [Cast and Attached Shadow Detection via Iterative Light and Geometry Reasoning](https://mlsanity.com/paper/2512.06179/) (2025)
- [WildHSR: Metric Feed-Forward 4D People-Scene Reconstruction from a 3D Foundation Model](https://mlsanity.com/paper/2609.29106/) (2026)

## Abstract (arXiv)

By analyzing the motion of people and other objects in a scene, we demonstrate how to infer depth, occlusion, lighting, and shadow information from video taken from a single camera viewpoint. This information is then used to composite new objects into the same scene with a high degree of automation and realism. In particular, when a user places a new object (2D cut-out) in the image, it is automatically rescaled, relit, occluded properly, and casts realistic shadows in the correct direction relative to the sun, and which conform properly to scene geometry. We demonstrate results (best viewed in supplementary video) on a range of scenes and compare to alternative methods for depth estimation and shadow compositing.