# PointAR: Efficient Lighting Estimation for Mobile Augmented Reality

arXiv: https://arxiv.org/abs/2004.00006 · HTML: https://mlsanity.com/paper/2004.00006/

## Key facts

- Cited by 19 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 30 Mar 2020.

## Built on by (most cited first, 19 of 19)

- [Learning Indoor Inverse Rendering with 3D Spatially-Varying Lighting](https://mlsanity.com/paper/2109.06061/) (2021)
- [Neural Light Field Estimation for Street Scenes with Differentiable Virtual Object Insertion](https://mlsanity.com/paper/2208.09480/) (2022)
- [EverLight: Indoor-Outdoor Editable HDR Lighting Estimation](https://mlsanity.com/paper/2304.13207/) (2023)
- [Photorealistic Object Insertion with Diffusion-Guided Inverse Rendering](https://mlsanity.com/paper/2408.09702/) (2024)
- [RGB-Depth Fusion GAN for Indoor Depth Completion](https://mlsanity.com/paper/2203.10856/) (2022)
- [NeRRF: 3D Reconstruction and View Synthesis for Transparent and Specular Objects with Neural Refractive-Reflective Fields](https://mlsanity.com/paper/2309.13039/) (2023)
- [LuxDiT: Lighting Estimation with Video Diffusion Transformer](https://mlsanity.com/paper/2509.03680/) (2025)
- [MegBA: A GPU-Based Distributed Library for Large-Scale Bundle Adjustment](https://mlsanity.com/paper/2112.01349/) (2021)
- [Editable Indoor Lighting Estimation](https://mlsanity.com/paper/2211.03928/) (2022)
- [Spatiotemporally Consistent HDR Indoor Lighting Estimation](https://mlsanity.com/paper/2305.04374/) (2023)
- [Lighting, Reflectance and Geometry Estimation from 360circ Panoramic Stereo](https://mlsanity.com/paper/2104.09886/) (2021)
- [RDFC-GAN: RGB-Depth Fusion CycleGAN for Indoor Depth Completion](https://mlsanity.com/paper/2306.03584/) (2023)
- [Arbitrary Point Cloud Upsampling with Spherical Mixture of Gaussians](https://mlsanity.com/paper/2208.05274/) (2022)
- [Xihe: A 3D Vision-based Lighting Estimation Framework for Mobile Augmented Reality](https://mlsanity.com/paper/2106.15280/) (2021)
- [Multi-Camera Lighting Estimation for Photorealistic Front-Facing Mobile Augmented Reality](https://mlsanity.com/paper/2301.06143/) (2023)
- [CleAR: Robust Context-Guided Generative Lighting Estimation for Mobile Augmented Reality](https://mlsanity.com/paper/2411.02179/) (2024)
- [OCHID-Fi: Occlusion-Robust Hand Pose Estimation in 3D via RF-Vision](https://mlsanity.com/paper/2308.10146/) (2023)
- [LitAR: Visually Coherent Lighting for Mobile Augmented Reality](https://mlsanity.com/paper/2301.06184/) (2023)
- [Leveraging Object Priors for Point Tracking](https://mlsanity.com/paper/2409.05786/) (2024)

## Abstract (arXiv)

We propose an efficient lighting estimation pipeline that is suitable to run on modern mobile devices, with comparable resource complexities to state-of-the-art mobile deep learning models. Our pipeline, PointAR, takes a single RGB-D image captured from the mobile camera and a 2D location in that image, and estimates 2nd order spherical harmonics coefficients. This estimated spherical harmonics coefficients can be directly utilized by rendering engines for supporting spatially variant indoor lighting, in the context of augmented reality. Our key insight is to formulate the lighting estimation as a point cloud-based learning problem directly from point clouds, which is in part inspired by the Monte Carlo integration leveraged by real-time spherical harmonics lighting. While existing approaches estimate lighting information with complex deep learning pipelines, our method focuses on reducing the computational complexity. Through both quantitative and qualitative experiments, we demonstrate that PointAR achieves lower lighting estimation errors compared to state-of-the-art methods. Further, our method requires an order of magnitude lower resource, comparable to that of mobile-specific DNNs.