# Weakly-supervised Single-view Image Relighting

arXiv: https://arxiv.org/abs/2303.13852 · HTML: https://mlsanity.com/paper/2303.13852/

## Key facts

- Cited by 14 papers in the map (data through 14 Oct 2026).
- Builds on 19 of its 44 references that are in the map.
- First posted on arXiv 24 Mar 2023.

## Builds on (most cited first, 19 of 19)

- [ShapeNet: An Information-Rich 3D Model Repository](https://mlsanity.com/paper/1512.03012/) (2015)
- [MegaDepth: Learning Single-View Depth Prediction from Internet Photos](https://mlsanity.com/paper/1804.00607/) (2018)
- [Inverse Rendering for Complex Indoor Scenes: Shape, Spatially-Varying Lighting and SVBRDF from a Single Image](https://mlsanity.com/paper/1905.02722/) (2019)
- [MoFA: Model-based Deep Convolutional Face Autoencoder for Unsupervised Monocular Reconstruction](https://mlsanity.com/paper/1703.10580/) (2017)
- [Single Image Portrait Relighting](https://mlsanity.com/paper/1905.00824/) (2019)
- [SfSNet: Learning Shape, Reflectance and Illuminance of Faces in the Wild](https://mlsanity.com/paper/1712.01261/) (2017)
- [InverseRenderNet: Learning single image inverse rendering](https://mlsanity.com/paper/1811.12328/) (2018)
- [CGIntrinsics: Better Intrinsic Image Decomposition through Physically-Based Rendering](https://mlsanity.com/paper/1808.08601/) (2018)
- [Neural Face Editing with Intrinsic Image Disentangling](https://mlsanity.com/paper/1704.04131/) (2017)
- [Inverse Path Tracing for Joint Material and Lighting Estimation](https://mlsanity.com/paper/1903.07145/) (2019)
- [Unsupervised Learning for Intrinsic Image Decomposition from a Single Image](https://mlsanity.com/paper/1911.09930/) (2019)
- [Self-supervised Outdoor Scene Relighting](https://mlsanity.com/paper/2107.03106/) (2021)
- [Self-Supervised Intrinsic Image Decomposition](https://mlsanity.com/paper/1711.03678/) (2017)
- [Learning Non-Lambertian Object Intrinsics across ShapeNet Categories](https://mlsanity.com/paper/1612.08510/) (2016)
- [Shape and Material Capture at Home](https://mlsanity.com/paper/2104.06397/) (2021)
- [Direct Intrinsics: Learning Albedo-Shading Decomposition by Convolutional Regression](https://mlsanity.com/paper/1512.02311/) (2015)
- [LIME: Live Intrinsic Material Estimation](https://mlsanity.com/paper/1801.01075/) (2018)
- [Faces as Lighting Probes via Unsupervised Deep Highlight Extraction](https://mlsanity.com/paper/1803.06340/) (2018)
- [Leveraging Multi-view Image Sets for Unsupervised Intrinsic Image Decomposition and Highlight Separation](https://mlsanity.com/paper/1911.07262/) (2019)

## Built on by (most cited first, 14 of 14)

- [IntrinsicAnything: Learning Diffusion Priors for Inverse Rendering Under Unknown Illumination](https://mlsanity.com/paper/2404.11593/) (2024)
- [IDArb: Intrinsic Decomposition for Arbitrary Number of Input Views and Illuminations](https://mlsanity.com/paper/2412.12083/) (2024)
- [GenLit: Reformulating Single-Image Relighting as Video Generation](https://mlsanity.com/paper/2412.11224/) (2024)
- [Neural LightRig: Unlocking Accurate Object Normal and Material Estimation with Multi-Light Diffusion](https://mlsanity.com/paper/2412.09593/) (2024)
- [Uni-Renderer: Unifying Rendering and Inverse Rendering Via Dual Stream Diffusion](https://mlsanity.com/paper/2412.15050/) (2024)
- [Diffusion Reflectance Map: Single-Image Stochastic Inverse Rendering of Illumination and Reflectance](https://mlsanity.com/paper/2312.04529/) (2023)
- [SuperMat: Physically Consistent PBR Material Estimation at Interactive Rates](https://mlsanity.com/paper/2411.17515/) (2024)
- [RENI++ A Rotation-Equivariant, Scale-Invariant, Natural Illumination Prior](https://mlsanity.com/paper/2311.09361/) (2023)
- [Relighting Scenes with Object Insertions in Neural Radiance Fields](https://mlsanity.com/paper/2406.14806/) (2024)
- [MERLiN: Single-Shot Material Estimation and Relighting for Photometric Stereo](https://mlsanity.com/paper/2409.00674/) (2024)
- [A General Albedo Recovery Approach for Aerial Photogrammetric Images through Inverse Rendering](https://mlsanity.com/paper/2409.03032/) (2024)
- [LumiTex: Towards High-Fidelity PBR Texture Generation with Illumination Context](https://mlsanity.com/paper/2511.19437/) (2025)
- [TokenLight: Precise Lighting Control in Images using Attribute Tokens](https://mlsanity.com/paper/2604.15310/) (2026)
- [WildRelight: A Real-World Benchmark and Physics-Guided Adaptation for Single-Image Relighting](https://mlsanity.com/paper/2605.11696/) (2026)

## Abstract (arXiv)

We present a learning-based approach to relight a single image of Lambertian and low-frequency specular objects. Our method enables inserting objects from photographs into new scenes and relighting them under the new environment lighting, which is essential for AR applications. To relight the object, we solve both inverse rendering and re-rendering. To resolve the ill-posed inverse rendering, we propose a weakly-supervised method by a low-rank constraint. To facilitate the weakly-supervised training, we contribute Relit, a large-scale (750K images) dataset of videos with aligned objects under changing illuminations. For re-rendering, we propose a differentiable specular rendering layer to render low-frequency non-Lambertian materials under various illuminations of spherical harmonics. The whole pipeline is end-to-end and efficient, allowing for a mobile app implementation of AR object insertion. Extensive evaluations demonstrate that our method achieves state-of-the-art performance. Project page: https://renjiaoyi.github.io/relighting/.