# Joint Shadow Generation and Relighting via Light-Geometry Interaction Maps

arXiv: https://arxiv.org/abs/2602.21820 · HTML: https://mlsanity.com/paper/2602.21820/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 26 of its 48 references that are in the map.
- First posted on arXiv 25 Feb 2026.

## Builds on (most cited first, 26 of 26)

- [High-Resolution Image Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2112.10752/) (2021)
- [The Unreasonable Effectiveness of Deep Features as a Perceptual Metric](https://mlsanity.com/paper/1801.03924/) (2018)
- [SDXL: Improving Latent Diffusion Models for High-Resolution Image Synthesis](https://mlsanity.com/paper/2307.01952/) (2023)
- [Depth Anything V2](https://mlsanity.com/paper/2406.09414/) (2024)
- [RGBleftrightarrowX: Image decomposition and synthesis using material- and lighting-aware diffusion models](https://mlsanity.com/paper/2405.00666/) (2024)
- [DiLightNet: Fine-grained Lighting Control for Diffusion-based Image Generation](https://mlsanity.com/paper/2402.11929/) (2024)
- [Mask-ShadowGAN: Learning to Remove Shadows from Unpaired Data](https://mlsanity.com/paper/1903.10683/) (2019)
- [Learning-based Inverse Rendering of Complex Indoor Scenes with Differentiable Monte Carlo Raytracing](https://mlsanity.com/paper/2211.03017/) (2022)
- [ObjectDrop: Bootstrapping Counterfactuals for Photorealistic Object Removal and Insertion](https://mlsanity.com/paper/2403.18818/) (2024)
- [SwitchLight: Co-design of Physics-driven Architecture and Pre-training Framework for Human Portrait Relighting](https://mlsanity.com/paper/2402.18848/) (2024)
- [LightIt: Illumination Modeling and Control for Diffusion Models](https://mlsanity.com/paper/2403.10615/) (2024)
- [LBM: Latent Bridge Matching for Fast Image-to-Image Translation](https://mlsanity.com/paper/2503.07535/) (2025)
- [Shadow Generation for Composite Image in Real-world Scenes](https://mlsanity.com/paper/2104.10338/) (2021)
- [OutCast: Outdoor Single-image Relighting with Cast Shadows](https://mlsanity.com/paper/2204.09341/) (2022)
- [IllumiNeRF: 3D Relighting Without Inverse Rendering](https://mlsanity.com/paper/2406.06527/) (2024)
- [VIDIT: Virtual Image Dataset for Illumination Transfer](https://mlsanity.com/paper/2005.05460/) (2020)
- [SSN: Soft Shadow Network for Image Compositing](https://mlsanity.com/paper/2007.08211/) (2020)
- [Controllable Shadow Generation Using Pixel Height Maps](https://mlsanity.com/paper/2207.05385/) (2022)
- [UrbanIR: Large-Scale Urban Scene Inverse Rendering from a Single Video](https://mlsanity.com/paper/2306.09349/) (2023)
- [Shadow Generation for Composite Image Using Diffusion model](https://mlsanity.com/paper/2403.15234/) (2024)
- [ZeroComp: Zero-shot Object Compositing from Image Intrinsics via Diffusion](https://mlsanity.com/paper/2410.08168/) (2024)
- [PixHt-Lab: Pixel Height Based Light Effect Generation for Image Compositing](https://mlsanity.com/paper/2303.00137/) (2023)
- [Shadow Generation with Decomposed Mask Prediction and Attentive Shadow Filling](https://mlsanity.com/paper/2306.17358/) (2023)
- [Retinex-Diffusion: On Controlling Illumination Conditions in Diffusion Models via Retinex Theory](https://mlsanity.com/paper/2407.20785/) (2024)
- [DESOBAv2: Towards Large-scale Real-world Dataset for Shadow Generation](https://mlsanity.com/paper/2308.09972/) (2023)
- [Controllable Shadow Generation with Single-Step Diffusion Models from Synthetic Data](https://mlsanity.com/paper/2412.11972/) (2024)

## Abstract (arXiv)

We propose Light-Geometry Interaction (LGI) maps, a novel representation that encodes light-aware occlusion from monocular depth. Unlike ray tracing, which requires full 3D reconstruction, LGI captures essential light-shadow interactions reliably and accurately, computed from off-the-shelf 2.5D depth map predictions. LGI explicitly ties illumination direction to geometry, providing a physics-inspired prior that constrains generative models. Without such prior, these models often produce floating shadows, inconsistent illumination, and implausible shadow geometry. Building on this representation, we propose a unified pipeline for joint shadow generation and relighting - unlike prior methods that treat them as disjoint tasks - capturing the intrinsic coupling of illumination and shadowing essential for modeling indirect effects. By embedding LGI into a bridge-matching generative backbone, we reduce ambiguity and enforce physically consistent light-shadow reasoning. To enable effective training, we curated the first large-scale benchmark dataset for joint shadow and relighting, covering reflections, transparency, and complex interreflections. Experiments show significant gains in realism and consistency across synthetic and real images. LGI thus bridges geometry-inspired rendering with generative modeling, enabling efficient, physically consistent shadow generation and relighting.