# Sparse Needlets for Lighting Estimation with Spherical Transport Loss

arXiv: https://arxiv.org/abs/2106.13090 · HTML: https://mlsanity.com/paper/2106.13090/

## Key facts

- Cited by 18 papers in the map (data through 14 Oct 2026).
- Builds on 32 of its 55 references that are in the map.
- First posted on arXiv 24 Jun 2021.

## Builds on (most cited first, 30 of 32)

- [Sinkhorn Distances: Lightspeed Computation of Optimal Transportation Distances](https://mlsanity.com/paper/1306.0895/) (2013)
- [Inverse Rendering for Complex Indoor Scenes: Shape, Spatially-Varying Lighting and SVBRDF from a Single Image](https://mlsanity.com/paper/1905.02722/) (2019)
- [Learning to Predict Indoor Illumination from a Single Image](https://mlsanity.com/paper/1704.00090/) (2017)
- [Optimal Entropy-Transport problems and a new Hellinger-Kantorovich distance between positive measures](https://mlsanity.com/paper/1508.07941/) (2015)
- [Smooth and Sparse Optimal Transport](https://mlsanity.com/paper/1710.06276/) (2017)
- [Deep Parametric Indoor Lighting Estimation](https://mlsanity.com/paper/1910.08812/) (2019)
- [Diverse Image Inpainting with Bidirectional and Autoregressive Transformers](https://mlsanity.com/paper/2104.12335/) (2021)
- [ESIR: End-to-end Scene Text Recognition via Iterative Image Rectification](https://mlsanity.com/paper/1812.05824/) (2018)
- [Fast Spatially-Varying Indoor Lighting Estimation](https://mlsanity.com/paper/1906.03799/) (2019)
- [Rendering Synthetic Objects into Legacy Photographs](https://mlsanity.com/paper/1912.11565/) (2019)
- [Spatial Fusion GAN for Image Synthesis](https://mlsanity.com/paper/1812.05840/) (2018)
- [DeepLight: Learning Illumination for Unconstrained Mobile Mixed Reality](https://mlsanity.com/paper/1904.01175/) (2019)
- [Lighthouse: Predicting Lighting Volumes for Spatially-Coherent Illumination](https://mlsanity.com/paper/2003.08367/) (2020)
- [Two-shot Spatially-varying BRDF and Shape Estimation](https://mlsanity.com/paper/2004.00403/) (2020)
- [Neural Illumination: Lighting Prediction for Indoor Environments](https://mlsanity.com/paper/1906.07370/) (2019)
- [Deep Outdoor Illumination Estimation](https://mlsanity.com/paper/1611.06403/) (2016)
- [EMLight: Lighting Estimation via Spherical Distribution Approximation](https://mlsanity.com/paper/2012.11116/) (2020)
- [A Dataset of Multi-Illumination Images in the Wild](https://mlsanity.com/paper/1910.08131/) (2019)
- [Verisimilar Image Synthesis for Accurate Detection and Recognition of Texts in Scenes](https://mlsanity.com/paper/1807.03021/) (2018)
- [All-Weather Deep Outdoor Lighting Estimation](https://mlsanity.com/paper/1906.04909/) (2019)
- [Unbalanced Feature Transport for Exemplar-based Image Translation](https://mlsanity.com/paper/2106.10482/) (2021)
- [GA-DAN: Geometry-Aware Domain Adaptation Network for Scene Text Detection and Recognition](https://mlsanity.com/paper/1907.09653/) (2019)
- [Scaling Algorithms for Unbalanced Transport Problems](https://mlsanity.com/paper/1607.05816/) (2016)
- [Intrinsic3D: High-Quality 3D Reconstruction by Joint Appearance and Geometry Optimization with Spatially-Varying Lighting](https://mlsanity.com/paper/1708.01670/) (2017)
- [GMLight: Lighting Estimation via Geometric Distribution Approximation](https://mlsanity.com/paper/2102.10244/) (2021)
- [Adversarial Image Composition with Auxiliary Illumination](https://mlsanity.com/paper/2009.08255/) (2020)
- [Accurate Scene Text Detection through Border Semantics Awareness and Bootstrapping](https://mlsanity.com/paper/1807.03547/) (2018)
- [Learning to Estimate Indoor Lighting from 3D Objects](https://mlsanity.com/paper/1806.03994/) (2018)
- [Scene Text Synthesis for Efficient and Effective Deep Network Training](https://mlsanity.com/paper/1901.09193/) (2019)
- [Spatial-Aware GAN for Unsupervised Person Re-identification](https://mlsanity.com/paper/1911.11312/) (2019)

## Built on by (most cited first, 18 of 18)

- [StyleLight: HDR Panorama Generation for Lighting Estimation and Editing](https://mlsanity.com/paper/2207.14811/) (2022)
- [Multimodal Image Synthesis and Editing: The Generative AI Era](https://mlsanity.com/paper/2112.13592/) (2021)
- [Blind Image Super-Resolution via Contrastive Representation Learning](https://mlsanity.com/paper/2107.00708/) (2021)
- [EverLight: Indoor-Outdoor Editable HDR Lighting Estimation](https://mlsanity.com/paper/2304.13207/) (2023)
- [Bi-level Feature Alignment for Versatile Image Translation and Manipulation](https://mlsanity.com/paper/2107.03021/) (2021)
- [VMRF: View Matching Neural Radiance Fields](https://mlsanity.com/paper/2207.02621/) (2022)
- [NeRRF: 3D Reconstruction and View Synthesis for Transparent and Specular Objects with Neural Refractive-Reflective Fields](https://mlsanity.com/paper/2309.13039/) (2023)
- [Accidental Light Probes](https://mlsanity.com/paper/2301.05211/) (2023)
- [Editable Indoor Lighting Estimation](https://mlsanity.com/paper/2211.03928/) (2022)
- [Spatiotemporally Consistent HDR Indoor Lighting Estimation](https://mlsanity.com/paper/2305.04374/) (2023)
- [Diffusion Reflectance Map: Single-Image Stochastic Inverse Rendering of Illumination and Reflectance](https://mlsanity.com/paper/2312.04529/) (2023)
- [Deep Graph Learning for Spatially-Varying Indoor Lighting Prediction](https://mlsanity.com/paper/2202.06300/) (2022)
- [Local-to-Global Panorama Inpainting for Locale-Aware Indoor Lighting Prediction](https://mlsanity.com/paper/2303.10344/) (2023)
- [MAIR++: Improving Multi-view Attention Inverse Rendering with Implicit Lighting Representation](https://mlsanity.com/paper/2408.06707/) (2024)
- [FBC-GAN: Diverse and Flexible Image Synthesis via Foreground-Background Composition](https://mlsanity.com/paper/2107.03166/) (2021)
- [MixLight: Borrowing the Best of both Spherical Harmonics and Gaussian Models](https://mlsanity.com/paper/2404.12768/) (2024)
- [Physically-Based Photometric Bundle Adjustment in Non-Lambertian Environments](https://mlsanity.com/paper/2409.11854/) (2024)
- [HDR Environment Map Estimation with Latent Diffusion Models](https://mlsanity.com/paper/2507.21261/) (2025)

## Abstract (arXiv)

Accurate lighting estimation is challenging yet critical to many computer vision and computer graphics tasks such as high-dynamic-range (HDR) relighting. Existing approaches model lighting in either frequency domain or spatial domain which is insufficient to represent the complex lighting conditions in scenes and tends to produce inaccurate estimation. This paper presents NeedleLight, a new lighting estimation model that represents illumination with needlets and allows lighting estimation in both frequency domain and spatial domain jointly. An optimal thresholding function is designed to achieve sparse needlets which trims redundant lighting parameters and demonstrates superior localization properties for illumination representation. In addition, a novel spherical transport loss is designed based on optimal transport theory which guides to regress lighting representation parameters with consideration of the spatial information. Furthermore, we propose a new metric that is concise yet effective by directly evaluating the estimated illumination maps rather than rendered images. Extensive experiments show that NeedleLight achieves superior lighting estimation consistently across multiple evaluation metrics as compared with state-of-the-art methods.