# Physically Controllable Relighting of Photographs

arXiv: https://arxiv.org/abs/2508.05626 · HTML: https://mlsanity.com/paper/2508.05626/

## Key facts

- Cited by 14 papers in the map (data through 14 Oct 2026).
- Builds on 25 of its 44 references that are in the map.
- First posted on arXiv 7 Aug 2025.

## Builds on (most cited first, 25 of 25)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Towards Robust Monocular Depth Estimation: Mixing Datasets for Zero-shot Cross-dataset Transfer](https://mlsanity.com/paper/1907.01341/) (2019)
- [Making Convolutional Networks Shift-Invariant Again](https://mlsanity.com/paper/1904.11486/) (2019)
- [Inverse Rendering for Complex Indoor Scenes: Shape, Spatially-Varying Lighting and SVBRDF from a Single Image](https://mlsanity.com/paper/1905.02722/) (2019)
- [RGBleftrightarrowX: Image decomposition and synthesis using material- and lighting-aware diffusion models](https://mlsanity.com/paper/2405.00666/) (2024)
- [Neural Inverse Rendering of an Indoor Scene from a Single Image](https://mlsanity.com/paper/1901.02453/) (2019)
- [DiLightNet: Fine-grained Lighting Control for Diffusion-based Image Generation](https://mlsanity.com/paper/2402.11929/) (2024)
- [Neural Gaffer: Relighting Any Object via Diffusion](https://mlsanity.com/paper/2406.07520/) (2024)
- [Learning-based Inverse Rendering of Complex Indoor Scenes with Differentiable Monte Carlo Raytracing](https://mlsanity.com/paper/2211.03017/) (2022)
- [Learning to Relight Portrait Images via a Virtual Light Stage and Synthetic-to-Real Adaptation](https://mlsanity.com/paper/2209.10510/) (2022)
- [IRISformer: Dense Vision Transformers for Single-Image Inverse Rendering in Indoor Scenes](https://mlsanity.com/paper/2206.08423/) (2022)
- [PPR10K: A Large-Scale Portrait Photo Retouching Dataset with Human-Region Mask and Group-Level Consistency](https://mlsanity.com/paper/2105.09180/) (2021)
- [Relightful Harmonization: Lighting-aware Portrait Background Replacement](https://mlsanity.com/paper/2312.06886/) (2023)
- [Intrinsic Image Decomposition via Ordinal Shading](https://mlsanity.com/paper/2311.12792/) (2023)
- [Free-viewpoint Indoor Neural Relighting from Multi-view Stereo](https://mlsanity.com/paper/2106.13299/) (2021)
- [Self-supervised Outdoor Scene Relighting](https://mlsanity.com/paper/2107.03106/) (2021)
- [Colorful Diffuse Intrinsic Image Decomposition in the Wild](https://mlsanity.com/paper/2409.13690/) (2024)
- [LumiNet: Latent Intrinsics Meets Diffusion Models for Indoor Scene Relighting](https://mlsanity.com/paper/2412.00177/) (2024)
- [A Diffusion Approach to Radiance Field Relighting using Multi-Illumination Synthesis](https://mlsanity.com/paper/2409.08947/) (2024)
- [OutCast: Outdoor Single-image Relighting with Cast Shadows](https://mlsanity.com/paper/2204.09341/) (2022)
- [ScribbleLight: Single Image Indoor Relighting with Scribbles](https://mlsanity.com/paper/2411.17696/) (2024)
- [GenLit: Reformulating Single-Image Relighting as Video Generation](https://mlsanity.com/paper/2412.11224/) (2024)
- [Outdoor inverse rendering from a single image using multiview self-supervision](https://mlsanity.com/paper/2102.06591/) (2021)
- [Latent Intrinsics Emerge from Training to Relight](https://mlsanity.com/paper/2405.21074/) (2024)
- [Intrinsic Harmonization for Illumination-Aware Compositing](https://mlsanity.com/paper/2312.03698/) (2023)

## Built on by (most cited first, 14 of 14)

- [Generative World Renderer](https://mlsanity.com/paper/2604.02329/) (2026)
- [LuxRemix: Lighting Decomposition and Remixing for Indoor Scenes](https://mlsanity.com/paper/2601.15283/) (2026)
- [GR3EN: Generative Relighting for 3D Environments](https://mlsanity.com/paper/2601.16272/) (2026)
- [MVID: Feed-Forward Multi-View Intrinsic Image Decomposition](https://mlsanity.com/paper/2512.23667/) (2025)
- [Relit-LiVE: Relight Video by Jointly Learning Environment Video](https://mlsanity.com/paper/2605.06658/) (2026)
- [Decoupled Illumination Priors for Spatially Controllable Multi-View Indoor Scene Relighting](https://mlsanity.com/paper/2607.08879/) (2026)
- [SyncLight: Single-Edit Multi-View Relighting](https://mlsanity.com/paper/2601.16981/) (2026)
- [LightMover: Generative Light Movement with Color and Intensity Controls](https://mlsanity.com/paper/2603.27209/) (2026)
- [PIXLRelight: Controllable Relighting via Intrinsic Conditioning](https://mlsanity.com/paper/2605.18735/) (2026)
- [TRON: Tracing Rays to Orchestrate a Neural Renderer for 3D Gaussian Reconstructions](https://mlsanity.com/paper/2606.11314/) (2026)
- [Lighting-Consistent Object Transfer Across Radiance Fields](https://mlsanity.com/paper/2606.22481/) (2026)
- [LightCrafter: PBR-Conditioned Video Diffusion Refinement for Controllable and Consistent Relighting](https://mlsanity.com/paper/2607.08016/) (2026)
- [Physically Based Rendering in the Latent Space](https://mlsanity.com/paper/2609.21054/) (2026)
- [RoomLight: A 2.5D Illumination Prior for Indoor Environments](https://mlsanity.com/paper/2609.28300/) (2026)

## Abstract (arXiv)

We present a self-supervised approach to in-the-wild image relighting that enables fully controllable, physically based illumination editing. We achieve this by combining the physical accuracy of traditional rendering with the photorealistic appearance made possible by neural rendering. Our pipeline works by inferring a colored mesh representation of a given scene using monocular estimates of geometry and intrinsic components. This representation allows users to define their desired illumination configuration in 3D. The scene under the new lighting can then be rendered using a path-tracing engine. We send this approximate rendering of the scene through a feed-forward neural renderer to predict the final photorealistic relighting result. We develop a differentiable rendering process to reconstruct in-the-wild scene illumination, enabling self-supervised training of our neural renderer on raw image collections. Our method represents a significant step in bringing the explicit physical control over lights available in typical 3D computer graphics tools, such as Blender, to in-the-wild relighting.