# HDR Environment Map Estimation for Real-Time Augmented Reality

arXiv: https://arxiv.org/abs/2011.10687 · HTML: https://mlsanity.com/paper/2011.10687/

## Key facts

- Cited by 47 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 21 Nov 2020.

## Built on by (most cited first, 30 of 47)

- [Editable Scene Simulation for Autonomous Driving via Collaborative LLM-Agents](https://mlsanity.com/paper/2402.05746/) (2024)
- [DiffusionLight: Light Probes for Free by Painting a Chrome Ball](https://mlsanity.com/paper/2312.09168/) (2023)
- [PanoDiffusion: 360-degree Panorama Outpainting via Diffusion](https://mlsanity.com/paper/2307.03177/) (2023)
- [Diverse Plausible 360-Degree Image Outpainting for Efficient 3DCG Background Creation](https://mlsanity.com/paper/2203.14668/) (2022)
- [Neural Light Field Estimation for Street Scenes with Differentiable Virtual Object Insertion](https://mlsanity.com/paper/2208.09480/) (2022)
- [CubeDiff: Repurposing Diffusion-Based Image Models for Panorama Generation](https://mlsanity.com/paper/2501.17162/) (2025)
- [Autoregressive Omni-Aware Outpainting for Open-Vocabulary 360-Degree Image Generation](https://mlsanity.com/paper/2309.03467/) (2023)
- [Guided Co-Modulated GAN for 360deg Field of View Extrapolation](https://mlsanity.com/paper/2204.07286/) (2022)
- [EverLight: Indoor-Outdoor Editable HDR Lighting Estimation](https://mlsanity.com/paper/2304.13207/) (2023)
- [A Survey of Representation Learning, Optimization Strategies, and Applications for Omnidirectional Vision](https://mlsanity.com/paper/2502.10444/) (2025)
- [3D Scene Generation: A Survey](https://mlsanity.com/paper/2505.05474/) (2025)
- [LightSim: Neural Lighting Simulation for Urban Scenes](https://mlsanity.com/paper/2312.06654/) (2023)
- [NeRRF: 3D Reconstruction and View Synthesis for Transparent and Specular Objects with Neural Refractive-Reflective Fields](https://mlsanity.com/paper/2309.13039/) (2023)
- [Cylin-Painting: Seamless 360textdegree Panoramic Image Outpainting and Beyond](https://mlsanity.com/paper/2204.08563/) (2022)
- [Editable Indoor Lighting Estimation](https://mlsanity.com/paper/2211.03928/) (2022)
- [Spatiotemporally Consistent HDR Indoor Lighting Estimation](https://mlsanity.com/paper/2305.04374/) (2023)
- [GaSLight: Gaussian Splats for Spatially-Varying Lighting in HDR](https://mlsanity.com/paper/2504.10809/) (2025)
- [DreamCube: 3D Panorama Generation via Multi-plane Synchronization](https://mlsanity.com/paper/2506.17206/) (2025)
- [DiffusionLight-Turbo: Accelerated Light Probes for Free via Single-Pass Chrome Ball Inpainting](https://mlsanity.com/paper/2507.01305/) (2025)
- [The Sky's the Limit: Re-lightable Outdoor Scenes via a Sky-pixel Constrained Illumination Prior and Outside-In Visibility](https://mlsanity.com/paper/2311.16937/) (2023)
- [OPa-Ma: Text Guided Mamba for 360-degree Image Out-painting](https://mlsanity.com/paper/2407.10923/) (2024)
- [360Anything: Geometry-Free Lifting of Images and Videos to 360deg](https://mlsanity.com/paper/2601.16192/) (2026)
- [RENI++ A Rotation-Equivariant, Scale-Invariant, Natural Illumination Prior](https://mlsanity.com/paper/2311.09361/) (2023)
- [Spatiotemporally Consistent Indoor Lighting Estimation with Diffusion Priors](https://mlsanity.com/paper/2508.08384/) (2025)
- [CamFreeDiff: Camera-free Image to Panorama Generation with Diffusion Model](https://mlsanity.com/paper/2407.07174/) (2024)
- [HORIZON: High-Resolution Semantically Controlled Panorama Synthesis](https://mlsanity.com/paper/2210.04522/) (2022)
- [Multi-Camera Lighting Estimation for Photorealistic Front-Facing Mobile Augmented Reality](https://mlsanity.com/paper/2301.06143/) (2023)
- [Local-to-Global Panorama Inpainting for Locale-Aware Indoor Lighting Prediction](https://mlsanity.com/paper/2303.10344/) (2023)
- [360U-Former: HDR Illumination Estimation with Panoramic Adapted Vision Transformers](https://mlsanity.com/paper/2410.13566/) (2024)
- [CleAR: Robust Context-Guided Generative Lighting Estimation for Mobile Augmented Reality](https://mlsanity.com/paper/2411.02179/) (2024)

## Abstract (arXiv)

We present a method to estimate an HDR environment map from a narrow field-of-view LDR camera image in real-time. This enables perceptually appealing reflections and shading on virtual objects of any material finish, from mirror to diffuse, rendered into a real physical environment using augmented reality. Our method is based on our efficient convolutional neural network architecture, EnvMapNet, trained end-to-end with two novel losses, ProjectionLoss for the generated image, and ClusterLoss for adversarial training. Through qualitative and quantitative comparison to state-of-the-art methods, we demonstrate that our algorithm reduces the directional error of estimated light sources by more than 50%, and achieves 3.7 times lower Frechet Inception Distance (FID). We further showcase a mobile application that is able to run our neural network model in under 9 ms on an iPhone XS, and render in real-time, visually coherent virtual objects in previously unseen real-world environments.