mlsanityData to 7 Oct 2026

DeepLight: Learning Illumination for Unconstrained Mobile Mixed Reality

arXiv 1904.01175 ↗First posted 2 Apr 2019cs.CV cs.GRcited by 63

Not on the map. Why?

Abstract (arXiv)

We present a learning-based method to infer plausible high dynamic range (HDR), omnidirectional illumination given an unconstrained, low dynamic range (LDR) image from a mobile phone camera with a limited field of view (FOV). For training data, we collect videos of various reflective spheres placed within the camera's FOV, leaving most of the background unoccluded, leveraging that materials with diverse reflectance functions reveal different lighting cues in a single exposure. We train a deep neural network to regress from the LDR background image to HDR lighting by matching the LDR ground truth sphere images to those rendered with the predicted illumination using image-based relighting, which is differentiable. Our inference runs at interactive frame rates on a mobile device, enabling realistic rendering of virtual objects into real scenes for mobile mixed reality. Training on automatically exposed and white-balanced videos, we improve the realism of rendered objects compared to the state-of-the art methods for both indoor and outdoor scenes.

Key facts. Cited by 63 papers in the map (data through 7 Oct 2026). First posted on arXiv 2 Apr 2019.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 11; 2022: 11; 2023: 16; 2024: 12; 2025: 10; 2026: 3.

63 papers.

Sort:

By area:

  1. 2024Editable Scene Simulation for Autonomous Driving via Collaborative LLM-Agentscited by 91
  2. 2023Neural Fields meet Explicit Geometric Representation for Inverse Rendering of Urban Scenescited by 89
  3. 2021Learning Indoor Inverse Rendering with 3D Spatially-Varying Lightingcited by 82
  4. 2023DiffusionLight: Light Probes for Free by Painting a Chrome Ballcited by 63
  5. 2022Physically-Based Editing of Indoor Scene Lighting from a Single Imagecited by 62
  6. 2022IRISformer: Dense Vision Transformers for Single-Image Inverse Rendering in Indoor Scenescited by 62
  7. 2022StyleLight: HDR Panorama Generation for Lighting Estimation and Editingcited by 59
  8. 2021Free-viewpoint Indoor Neural Relighting from Multi-view Stereocited by 50
  9. 2022Diverse Plausible 360-Degree Image Outpainting for Efficient 3DCG Background Creationcited by 47
  10. 2022Relighting4D: Neural Relightable Human from Videoscited by 41
  11. 2022Neural Light Field Estimation for Street Scenes with Differentiable Virtual Object Insertioncited by 35
  12. 2024MeshAvatar: Learning High-quality Triangular Human Avatars from Multi-view Videoscited by 34
38 more
  1. 2023EverLight: Indoor-Outdoor Editable HDR Lighting Estimationcited by 32
  2. 2022OutCast: Outdoor Single-image Relighting with Cast Shadowscited by 30
  3. 2021GMLight: Lighting Estimation via Geometric Distribution Approximationcited by 28
  4. 2024Photorealistic Object Insertion with Diffusion-Guided Inverse Renderingcited by 27
  5. 2023LightSim: Neural Lighting Simulation for Urban Scenescited by 20
  6. 2021Sparse Needlets for Lighting Estimation with Spherical Transport Losscited by 18
  7. 2022SPIDR: SDF-based Neural Point Fields for Illumination and Deformationcited by 17
  8. 2021Spatially-Varying Outdoor Lighting Estimation from Intrinsicscited by 14
  9. 2023Eclipse: Disambiguating Illumination and Materials using Unintended Shadowscited by 14
  10. 2025LuxDiT: Lighting Estimation with Video Diffusion Transformercited by 14
  11. 2023Accidental Light Probescited by 13
  12. 2024Generative Multiview Relighting for 3D Reconstruction under Extreme Illumination Variationcited by 13
  13. 2022Editable Indoor Lighting Estimationcited by 12
  14. 2023Spatiotemporally Consistent HDR Indoor Lighting Estimationcited by 12
  15. 2021Lighting, Reflectance and Geometry Estimation from 360circ Panoramic Stereocited by 11
  16. 2023Controllable Light Diffusion for Portraitscited by 11
  17. 2023Diffusion Reflectance Map: Single-Image Stochastic Inverse Rendering of Illumination and Reflectancecited by 11
  18. 2025DiffusionLight-Turbo: Accelerated Light Probes for Free via Single-Pass Chrome Ball Inpaintingcited by 11
  19. 2025DNF-Intrinsic: Deterministic Noise-Free Diffusion for Indoor Inverse Renderingcited by 11
  20. 2022nLMVS-Net: Deep Non-Lambertian Multi-View Stereocited by 10
  21. 2025GCC: Generative Color Constancy via Diffusing a Color Checkercited by 8
  22. 2023Differentiable Visual Computing for Inverse Problems and Machine Learningcited by 7
  23. 2024OPa-Ma: Text Guided Mamba for 360-degree Image Out-paintingcited by 7
  24. 2021Repopulating Street Scenescited by 6
  25. 2023Virtual Occlusions Through Implicit Depthcited by 5
  26. 2023A Real-time Method for Inserting Virtual Objects into Neural Radiance Fieldscited by 5
  27. 2023Virtual Home Staging: Inverse Rendering and Editing an Indoor Panorama under Natural Illuminationcited by 5
  28. 2023Towards a Perceptual Evaluation Framework for Lighting Estimationcited by 5
  29. 2025Spatiotemporally Consistent Indoor Lighting Estimation with Diffusion Priorscited by 5
  30. 2023Local-to-Global Panorama Inpainting for Locale-Aware Indoor Lighting Predictioncited by 3
  31. 2024LightOctree: Lightweight 3D Spatially-Coherent Indoor Lighting Estimationcited by 3
  32. 2024360U-Former: HDR Illumination Estimation with Panoramic Adapted Vision Transformerscited by 3
  33. 2024CleAR: Robust Context-Guided Generative Lighting Estimation for Mobile Augmented Realitycited by 3
  34. 2025UniLight: A Unified Representation for Lightingcited by 3
  35. 2021Beyond Visual Attractiveness: Physically Plausible Single Image HDR Reconstruction for Spherical Panoramascited by 2
  36. 2021SITTA: Single Image Texture Translation for Data Augmentationcited by 2
  37. 2021Spatially and color consistent environment lighting estimation using deep neural networks for mixed realitycited by 1
  38. 2024Physically-Based Photometric Bundle Adjustment in Non-Lambertian Environmentscited by 1

50 most cited of 63 All 63 (JSON)