# Self-supervised Outdoor Scene Relighting

arXiv: https://arxiv.org/abs/2107.03106 · HTML: https://mlsanity.com/paper/2107.03106/

## Key facts

- Cited by 48 papers in the map (data through 14 Oct 2026).
- Builds on 21 of its 57 references that are in the map.
- First posted on arXiv 7 Jul 2021.

## Builds on (most cited first, 21 of 21)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [U-Net: Convolutional Networks for Biomedical Image Segmentation](https://mlsanity.com/paper/1505.04597/) (2015)
- [Very Deep Convolutional Networks for Large-Scale Image Recognition](https://mlsanity.com/paper/1409.1556/) (2014)
- [Pyramid Scene Parsing Network](https://mlsanity.com/paper/1612.01105/) (2016)
- [Conditional Generative Adversarial Nets](https://mlsanity.com/paper/1411.1784/) (2014)
- [Semantic Image Synthesis with Spatially-Adaptive Normalization](https://mlsanity.com/paper/1903.07291/) (2019)
- [Least Squares Generative Adversarial Networks](https://mlsanity.com/paper/1611.04076/) (2016)
- [MegaDepth: Learning Single-View Depth Prediction from Internet Photos](https://mlsanity.com/paper/1804.00607/) (2018)
- [Deferred Neural Rendering: Image Synthesis using Neural Textures](https://mlsanity.com/paper/1904.12356/) (2019)
- [Deep Photo Style Transfer](https://mlsanity.com/paper/1703.07511/) (2017)
- [Neural Rerendering in the Wild](https://mlsanity.com/paper/1904.04290/) (2019)
- [Neural Inverse Rendering of an Indoor Scene from a Single Image](https://mlsanity.com/paper/1901.02453/) (2019)
- [InverseRenderNet: Learning single image inverse rendering](https://mlsanity.com/paper/1811.12328/) (2018)
- [Learning Intrinsic Image Decomposition from Watching the World](https://mlsanity.com/paper/1804.00582/) (2018)
- [Fast Spatially-Varying Indoor Lighting Estimation](https://mlsanity.com/paper/1906.03799/) (2019)
- [Rendering Synthetic Objects into Legacy Photographs](https://mlsanity.com/paper/1912.11565/) (2019)
- [Neural Face Editing with Intrinsic Image Disentangling](https://mlsanity.com/paper/1704.04131/) (2017)
- [Deep Outdoor Illumination Estimation](https://mlsanity.com/paper/1611.06403/) (2016)
- [LIME: Live Intrinsic Material Estimation](https://mlsanity.com/paper/1801.01075/) (2018)
- [LookinGood: Enhancing Performance Capture with Real-time Neural Re-Rendering](https://mlsanity.com/paper/1811.05029/) (2018)
- [End-to-End Time-Lapse Video Synthesis from a Single Outdoor Image](https://mlsanity.com/paper/1904.00680/) (2019)

## Built on by (most cited first, 30 of 48)

- [NeRO: Neural Geometry and BRDF Reconstruction of Reflective Objects from Multiview Images](https://mlsanity.com/paper/2305.17398/) (2023)
- [RGBleftrightarrowX: Image decomposition and synthesis using material- and lighting-aware diffusion models](https://mlsanity.com/paper/2405.00666/) (2024)
- [NeRF for Outdoor Scene Relighting](https://mlsanity.com/paper/2112.05140/) (2021)
- [DiLightNet: Fine-grained Lighting Control for Diffusion-based Image Generation](https://mlsanity.com/paper/2402.11929/) (2024)
- [Neural Gaffer: Relighting Any Object via Diffusion](https://mlsanity.com/paper/2406.07520/) (2024)
- [IntrinsicAnything: Learning Diffusion Priors for Inverse Rendering Under Unknown Illumination](https://mlsanity.com/paper/2404.11593/) (2024)
- [LightIt: Illumination Modeling and Control for Diffusion Models](https://mlsanity.com/paper/2403.10615/) (2024)
- [Stanford-ORB: A Real-World 3D Object Inverse Rendering Benchmark](https://mlsanity.com/paper/2310.16044/) (2023)
- [LumiNet: Latent Intrinsics Meets Diffusion Models for Indoor Scene Relighting](https://mlsanity.com/paper/2412.00177/) (2024)
- [A Diffusion Approach to Radiance Field Relighting using Multi-Illumination Synthesis](https://mlsanity.com/paper/2409.08947/) (2024)
- [De-rendering 3D Objects in the Wild](https://mlsanity.com/paper/2201.02279/) (2022)
- [OutCast: Outdoor Single-image Relighting with Cast Shadows](https://mlsanity.com/paper/2204.09341/) (2022)
- [ScribbleLight: Single Image Indoor Relighting with Scribbles](https://mlsanity.com/paper/2411.17696/) (2024)
- [UrbanIR: Large-Scale Urban Scene Inverse Rendering from a Single Video](https://mlsanity.com/paper/2306.09349/) (2023)
- [LumiGauss: Relightable Gaussian Splatting in the Wild](https://mlsanity.com/paper/2408.04474/) (2024)
- [LightSim: Neural Lighting Simulation for Urban Scenes](https://mlsanity.com/paper/2312.06654/) (2023)
- [GenLit: Reformulating Single-Image Relighting as Video Generation](https://mlsanity.com/paper/2412.11224/) (2024)
- [Towers of Babel: Combining Images, Language, and 3D Geometry for Learning Multimodal Vision](https://mlsanity.com/paper/2108.05863/) (2021)
- [Latent Intrinsics Emerge from Training to Relight](https://mlsanity.com/paper/2405.21074/) (2024)
- [IntrinsicEdit: Precise generative image manipulation in intrinsic space](https://mlsanity.com/paper/2505.08889/) (2025)
- [Factorized and Controllable Neural Re-Rendering of Outdoor Scene for Photo Extrapolation](https://mlsanity.com/paper/2207.06899/) (2022)
- [Ouroboros: Single-step Diffusion Models for Cycle-consistent Forward and Inverse Rendering](https://mlsanity.com/paper/2508.14461/) (2025)
- [Weakly-supervised Single-view Image Relighting](https://mlsanity.com/paper/2303.13852/) (2023)
- [Physically Controllable Relighting of Photographs](https://mlsanity.com/paper/2508.05626/) (2025)
- [Seeing a Rose in Five Thousand Ways](https://mlsanity.com/paper/2212.04965/) (2022)
- [Rotation-Equivariant Conditional Spherical Neural Fields for Learning a Natural Illumination Prior](https://mlsanity.com/paper/2206.03858/) (2022)
- [Inverse Rendering of Translucent Objects using Physical and Neural Renderers](https://mlsanity.com/paper/2305.08336/) (2023)
- [LuxRemix: Lighting Decomposition and Remixing for Indoor Scenes](https://mlsanity.com/paper/2601.15283/) (2026)
- [GAN2X: Non-Lambertian Inverse Rendering of Image GANs](https://mlsanity.com/paper/2206.09244/) (2022)
- [MV-CoLight: Efficient Object Compositing with Consistent Lighting and Shadow Generation](https://mlsanity.com/paper/2505.21483/) (2025)

## Abstract (arXiv)

Outdoor scene relighting is a challenging problem that requires good understanding of the scene geometry, illumination and albedo. Current techniques are completely supervised, requiring high quality synthetic renderings to train a solution. Such renderings are synthesized using priors learned from limited data. In contrast, we propose a self-supervised approach for relighting. Our approach is trained only on corpora of images collected from the internet without any user-supervision. This virtually endless source of training data allows training a general relighting solution. Our approach first decomposes an image into its albedo, geometry and illumination. A novel relighting is then produced by modifying the illumination parameters. Our solution capture shadow using a dedicated shadow prediction map, and does not rely on accurate geometry estimation. We evaluate our technique subjectively and objectively using a new dataset with ground-truth relighting. Results show the ability of our technique to produce photo-realistic and physically plausible results, that generalizes to unseen scenes.