# Automatic Scene Inference for 3D Object Compositing

arXiv: https://arxiv.org/abs/1912.12297 · HTML: https://mlsanity.com/paper/1912.12297/

## Key facts

- Cited by 36 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 24 Dec 2019.

## Built on by (most cited first, 30 of 36)

- [Learning-based Inverse Rendering of Complex Indoor Scenes with Differentiable Monte Carlo Raytracing](https://mlsanity.com/paper/2211.03017/) (2022)
- [DiffusionLight: Light Probes for Free by Painting a Chrome Ball](https://mlsanity.com/paper/2312.09168/) (2023)
- [Making Images Real Again: A Comprehensive Survey on Deep Image Composition](https://mlsanity.com/paper/2106.14490/) (2021)
- [Physically-Based Editing of Indoor Scene Lighting from a Single Image](https://mlsanity.com/paper/2205.09343/) (2022)
- [Colorful Diffuse Intrinsic Image Decomposition in the Wild](https://mlsanity.com/paper/2409.13690/) (2024)
- [ObjectStitch: Generative Object Compositing](https://mlsanity.com/paper/2212.00932/) (2022)
- [Shadow Generation for Composite Image in Real-world Scenes](https://mlsanity.com/paper/2104.10338/) (2021)
- [CTRL-C: Camera calibration TRansformer with Line-Classification](https://mlsanity.com/paper/2109.02259/) (2021)
- [Factorized Inverse Path Tracing for Efficient and Accurate Material-Lighting Estimation](https://mlsanity.com/paper/2304.05669/) (2023)
- [PhotoScene: Photorealistic Material and Lighting Transfer for Indoor Scenes](https://mlsanity.com/paper/2207.00757/) (2022)
- [Shadow Generation for Composite Image Using Diffusion model](https://mlsanity.com/paper/2403.15234/) (2024)
- [Latent Intrinsics Emerge from Training to Relight](https://mlsanity.com/paper/2405.21074/) (2024)
- [ZeroComp: Zero-shot Object Compositing from Image Intrinsics via Diffusion](https://mlsanity.com/paper/2410.08168/) (2024)
- [Cross-domain Compositing with Pretrained Diffusion Models](https://mlsanity.com/paper/2302.10167/) (2023)
- [Spatiotemporally Consistent HDR Indoor Lighting Estimation](https://mlsanity.com/paper/2305.04374/) (2023)
- [DiffusionLight-Turbo: Accelerated Light Probes for Free via Single-Pass Chrome Ball Inpainting](https://mlsanity.com/paper/2507.01305/) (2025)
- [Deep Image-based Illumination Harmonization](https://mlsanity.com/paper/2108.00150/) (2021)
- [Human Image Generation: A Comprehensive Survey](https://mlsanity.com/paper/2212.08896/) (2022)
- [Intrinsic Harmonization for Illumination-Aware Compositing](https://mlsanity.com/paper/2312.03698/) (2023)
- [Mobile Augmented Reality: User Interfaces, Frameworks, and Intelligence](https://mlsanity.com/paper/2106.08710/) (2021)
- [Navigating Beyond Instructions: Vision-and-Language Navigation in Obstructed Environments](https://mlsanity.com/paper/2407.21452/) (2024)
- [Conditional 360-degree Image Synthesis for Immersive Indoor Scene Decoration](https://mlsanity.com/paper/2307.09621/) (2023)
- [Repopulating Street Scenes](https://mlsanity.com/paper/2103.16183/) (2021)
- [SAC-GAN: Structure-Aware Image Composition](https://mlsanity.com/paper/2112.06596/) (2021)
- [A Real-time Method for Inserting Virtual Objects into Neural Radiance Fields](https://mlsanity.com/paper/2310.05837/) (2023)
- [LightOctree: Lightweight 3D Spatially-Coherent Indoor Lighting Estimation](https://mlsanity.com/paper/2404.03925/) (2024)
- [Local-to-Global Panorama Inpainting for Locale-Aware Indoor Lighting Prediction](https://mlsanity.com/paper/2303.10344/) (2023)
- [Animating Street View](https://mlsanity.com/paper/2310.08534/) (2023)
- [Refracting Reality: Generating Images with Realistic Transparent Objects](https://mlsanity.com/paper/2511.17340/) (2025)
- [Neural Scene Decoration from a Single Photograph](https://mlsanity.com/paper/2108.01806/) (2021)

## Abstract (arXiv)

We present a user-friendly image editing system that supports a drag-and-drop object insertion (where the user merely drags objects into the image, and the system automatically places them in 3D and relights them appropriately), post-process illumination editing, and depth-of-field manipulation. Underlying our system is a fully automatic technique for recovering a comprehensive 3D scene model (geometry, illumination, diffuse albedo and camera parameters) from a single, low dynamic range photograph. This is made possible by two novel contributions: an illumination inference algorithm that recovers a full lighting model of the scene (including light sources that are not directly visible in the photograph), and a depth estimation algorithm that combines data-driven depth transfer with geometric reasoning about the scene layout. A user study shows that our system produces perceptually convincing results, and achieves the same level of realism as techniques that require significant user interaction.