mlsanityData to 14 Oct 2026

Local Relighting of Real Scenes

arXiv 2207.02774 ↗First posted 6 Jul 2022cs.CV cs.GRcited by 2

Not on the map. Why?

Abstract (arXiv)

We introduce the task of local relighting, which changes a photograph of a scene by switching on and off the light sources that are visible within the image. This new task differs from the traditional image relighting problem, as it introduces the challenge of detecting light sources and inferring the pattern of light that emanates from them. We propose an approach for local relighting that trains a model without supervision of any novel image dataset by using synthetically generated image pairs from another model. Concretely, we collect paired training images from a stylespace-manipulated GAN; then we use these images to train a conditional image-to-image model. To benchmark local relighting, we introduce Lonoff, a collection of 306 precisely aligned images taken in indoor spaces with different combinations of lights switched on. We show that our method significantly outperforms baseline methods based on GAN inversion. Finally, we demonstrate extensions of our method that control different light sources separately. We invite the community to tackle this new task of local relighting.

Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Builds on 24 of its 33 references that are in the map. First posted on arXiv 6 Jul 2022.

Builds on

24 of 33 references.

References · 24

  1. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  2. 2018The Unreasonable Effectiveness of Deep Features as a Perceptual Metriccited by 11,177
  3. 2018A Style-Based Generator Architecture for Generative Adversarial Networkscited by 6,013
  4. 2016Image-to-Image Translation with Conditional Adversarial Networkscited by 4,527
  5. 2019Analyzing and Improving the Image Quality of StyleGANcited by 3,588
  6. 2018Unified Perceptual Parsing for Scene Understandingcited by 1,271
  7. 2020Training Generative Adversarial Networks with Limited Datacited by 1,186
  8. 2017High-Resolution Image Synthesis and Semantic Manipulation with Conditional GANscited by 1,179
  9. 2019Interpreting the Latent Space of GANs for Semantic Face Editingcited by 695
  10. 2020GANSpace: Discovering Interpretable GAN Controlscited by 614
  11. 2021Designing an Encoder for StyleGAN Image Manipulationcited by 544
  12. 2020In-Domain GAN Inversion for Real Image Editingcited by 420
12 more
  1. 2020StyleSpace Analysis: Disentangled Controls for StyleGAN Image Generationcited by 332
  2. 2021DatasetGAN: Efficient Labeled Data Factory with Minimal Human Effortcited by 264
  3. 2019On the "steerability" of generative adversarial networkscited by 248
  4. 2019Image Processing Using Multi-Code GAN Priorcited by 185
  5. 2016Learning-Based View Synthesis for Light Field Camerascited by 145
  6. 2020StyleGAN2 Distillation for Feed-forward Image Manipulationcited by 78
  7. 2020VIDIT: Virtual Image Dataset for Illumination Transfercited by 26
  8. 2021NTIRE 2021 Depth Guided Image Relighting Challengecited by 23
  9. 20202D Image Relighting with Image-to-Image Translationcited by 10
  10. 2021Multi-modal Bifurcated Network for Depth Guided Image Relightingcited by 6
  11. 2021Multi-scale Self-calibrated Network for Image Light Source Transfercited by 4
  12. 2021Physically Inspired Dense Fusion Networks for Relightingcited by 3

Built on by

Cited by year

Citing papers per year: 2023: 2.

2 papers.

  1. 2023Open Problems and Fundamental Limitations of Reinforcement Learning from Human Feedbackcited by 759
  2. 2023Does Localization Inform Editing? Surprising Differences in Causality-Based Localization vs. Knowledge Editing in Language Modelscited by 243