# SIRfyN: Single Image Relighting from your Neighbors

arXiv: https://arxiv.org/abs/2112.04497 · HTML: https://mlsanity.com/paper/2112.04497/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 30 of its 67 references that are in the map.
- First posted on arXiv 8 Dec 2021.

## Builds on (most cited first, 30 of 30)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [A Simple Framework for Contrastive Learning of Visual Representations](https://mlsanity.com/paper/2002.05709/) (2020)
- [Image-to-Image Translation with Conditional Adversarial Networks](https://mlsanity.com/paper/1611.07004/) (2016)
- [LSUN: Construction of a Large-scale Image Dataset using Deep Learning with Humans in the Loop](https://mlsanity.com/paper/1506.03365/) (2015)
- [Group Equivariant Convolutional Networks](https://mlsanity.com/paper/1602.07576/) (2016)
- [PhySG: Inverse Rendering with Spherical Gaussians for Physics-based Material Editing and Relighting](https://mlsanity.com/paper/2104.00674/) (2021)
- [Learning Category-Specific Mesh Reconstruction from Image Collections](https://mlsanity.com/paper/1803.07549/) (2018)
- [Understanding image representations by measuring their equivariance and equivalence](https://mlsanity.com/paper/1411.5908/) (2014)
- [Geometric GAN](https://mlsanity.com/paper/1705.02894/) (2017)
- [Inverse Rendering for Complex Indoor Scenes: Shape, Spatially-Varying Lighting and SVBRDF from a Single Image](https://mlsanity.com/paper/1905.02722/) (2019)
- [Multi-view Supervision for Single-view Reconstruction via Differentiable Ray Consistency](https://mlsanity.com/paper/1704.06254/) (2017)
- [Effectively Unbiased FID and Inception Score and where to find them](https://mlsanity.com/paper/1911.07023/) (2019)
- [MoFA: Model-based Deep Convolutional Face Autoencoder for Unsupervised Monocular Reconstruction](https://mlsanity.com/paper/1703.10580/) (2017)
- [Single Image Portrait Relighting](https://mlsanity.com/paper/1905.00824/) (2019)
- [InverseRenderNet: Learning single image inverse rendering](https://mlsanity.com/paper/1811.12328/) (2018)
- [Learning Indoor Inverse Rendering with 3D Spatially-Varying Lighting](https://mlsanity.com/paper/2109.06061/) (2021)
- [Learning Intrinsic Image Decomposition from Watching the World](https://mlsanity.com/paper/1804.00582/) (2018)
- [VIDIT: Virtual Image Dataset for Illumination Transfer](https://mlsanity.com/paper/2005.05460/) (2020)
- [SSN: Soft Shadow Network for Image Compositing](https://mlsanity.com/paper/2007.08211/) (2020)
- [NTIRE 2021 Depth Guided Image Relighting Challenge](https://mlsanity.com/paper/2104.13365/) (2021)
- [AIM 2020: Scene Relighting and Illumination Estimation Challenge](https://mlsanity.com/paper/2009.12798/) (2020)
- [Intrinsic Image Decomposition using Paradigms](https://mlsanity.com/paper/2011.10512/) (2020)
- [People as Scene Probes](https://mlsanity.com/paper/2007.09209/) (2020)
- [WDRN : A Wavelet Decomposed RelightNet for Image Relighting](https://mlsanity.com/paper/2009.06678/) (2020)
- [Relighting Images in the Wild with a Self-Supervised Siamese Auto-Encoder](https://mlsanity.com/paper/2012.06444/) (2020)
- [Deep Relighting Networks for Image Light Source Manipulation](https://mlsanity.com/paper/2008.08298/) (2020)
- [2D Image Relighting with Image-to-Image Translation](https://mlsanity.com/paper/2006.07816/) (2020)
- [Multi-modal Bifurcated Network for Depth Guided Image Relighting](https://mlsanity.com/paper/2105.00690/) (2021)
- [SILT: Self-supervised Lighting Transfer Using Implicit Image Decomposition](https://mlsanity.com/paper/2110.12914/) (2021)
- [Controllable Data Augmentation Through Deep Relighting](https://mlsanity.com/paper/2110.13996/) (2021)

## Built on by (most cited first, 1 of 1)

- [Relighting from a Single Image: Datasets and Deep Intrinsic-based Architecture](https://mlsanity.com/paper/2409.18770/) (2024)

## Abstract (arXiv)

We show how to relight a scene, depicted in a single image, such that (a) the overall shading has changed and (b) the resulting image looks like a natural image of that scene. Applications for such a procedure include generating training data and building authoring environments. Naive methods for doing this fail. One reason is that shading and albedo are quite strongly related; for example, sharp boundaries in shading tend to appear at depth discontinuities, which usually apparent in albedo. The same scene can be lit in different ways, and established theory shows the different lightings form a cone (the illumination cone). Novel theory shows that one can use similar scenes to estimate the different lightings that apply to a given scene, with bounded expected error. Our method exploits this theory to estimate a representation of the available lighting fields in the form of imputed generators of the illumination cone. Our procedure does not require expensive "inverse graphics" datasets, and sees no ground truth data of any kind. Qualitative evaluation suggests the method can erase and restore soft indoor shadows, and can "steer" light around a scene. We offer a summary quantitative evaluation of the method with a novel application of the FID. An extension of the FID allows per-generated-image evaluation. Furthermore, we offer qualitative evaluation with a user study, and show that our method produces images that can successfully be used for data augmentation.