# VENI: Variational Encoder for Natural Illumination

arXiv: https://arxiv.org/abs/2601.14079 · HTML: https://mlsanity.com/paper/2601.14079/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 37 of its 57 references that are in the map.
- First posted on arXiv 20 Jan 2026.

## Builds on (most cited first, 30 of 37)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [A Style-Based Generator Architecture for Generative Adversarial Networks](https://mlsanity.com/paper/1812.04948/) (2018)
- [DeepSDF: Learning Continuous Signed Distance Functions for Shape Representation](https://mlsanity.com/paper/1901.05103/) (2019)
- [Depth Map Prediction from a Single Image using a Multi-Scale Deep Network](https://mlsanity.com/paper/1406.2283/) (2014)
- [Geometric Deep Learning: Grids, Groups, Graphs, Geodesics, and Gauges](https://mlsanity.com/paper/2104.13478/) (2021)
- [Tensor field networks: Rotation- and translation-equivariant neural networks for 3D point clouds](https://mlsanity.com/paper/1802.08219/) (2018)
- [MegaDepth: Learning Single-View Depth Prediction from Internet Photos](https://mlsanity.com/paper/1804.00607/) (2018)
- [SE(3)-Transformers: 3D Roto-Translation Equivariant Attention Networks](https://mlsanity.com/paper/2006.10503/) (2020)
- [Depth Pro: Sharp Monocular Metric Depth in Less Than a Second](https://mlsanity.com/paper/2410.02073/) (2024)
- [3D Steerable CNNs: Learning Rotationally Equivariant Features in Volumetric Data](https://mlsanity.com/paper/1807.02547/) (2018)
- [PhySG: Inverse Rendering with Spherical Gaussians for Physics-based Material Editing and Relighting](https://mlsanity.com/paper/2104.00674/) (2021)
- [Steerable CNNs](https://mlsanity.com/paper/1612.08498/) (2016)
- [Vector Neurons: A General Framework for SO(3)-Equivariant Networks](https://mlsanity.com/paper/2104.12229/) (2021)
- [Neural-PIL: Neural Pre-Integrated Lighting for Reflectance Decomposition](https://mlsanity.com/paper/2110.14373/) (2021)
- [Modeling Indirect Illumination for Inverse Rendering](https://mlsanity.com/paper/2204.06837/) (2022)
- [HDR image reconstruction from a single exposure using deep CNNs](https://mlsanity.com/paper/1710.07480/) (2017)
- [Learning to Predict Indoor Illumination from a Single Image](https://mlsanity.com/paper/1704.00090/) (2017)
- [PanoFormer: Panorama Transformer for Indoor 360 Depth Estimation](https://mlsanity.com/paper/2203.09283/) (2022)
- [Learning Indoor Inverse Rendering with 3D Spatially-Varying Lighting](https://mlsanity.com/paper/2109.06061/) (2021)
- [DiffusionLight: Light Probes for Free by Painting a Chrome Ball](https://mlsanity.com/paper/2312.09168/) (2023)
- [StyleLight: HDR Panorama Generation for Lighting Estimation and Editing](https://mlsanity.com/paper/2207.14811/) (2022)
- [EMLight: Lighting Estimation via Spherical Distribution Approximation](https://mlsanity.com/paper/2012.11116/) (2020)
- [SPAR3D: Stable Point-Aware Reconstruction of 3D Objects from Single Images](https://mlsanity.com/paper/2501.04689/) (2025)
- [EverLight: Indoor-Outdoor Editable HDR Lighting Estimation](https://mlsanity.com/paper/2304.13207/) (2023)
- [VN-Transformer: Rotation-Equivariant Attention for Vector Neurons](https://mlsanity.com/paper/2206.04176/) (2022)
- [Dist-GAN: An Improved GAN using Distance Constraints](https://mlsanity.com/paper/1803.08887/) (2018)
- [LEDiff: Latent Exposure Diffusion for HDR Generation](https://mlsanity.com/paper/2412.14456/) (2024)
- [Outdoor inverse rendering from a single image using multiview self-supervision](https://mlsanity.com/paper/2102.06591/) (2021)
- [Luminance Attentive Networks for HDR Image and Panorama Reconstruction](https://mlsanity.com/paper/2109.06688/) (2021)

## Built on by (most cited first, 1 of 1)

- [RoomLight: A 2.5D Illumination Prior for Indoor Environments](https://mlsanity.com/paper/2609.28300/) (2026)

## Abstract (arXiv)

Inverse rendering is an ill-posed problem, but priors such as illumination priors can help simplify it. Existing work either disregards the spherical and rotation-equivariant nature of illumination environments or does not provide a well-behaved latent space. We propose a rotation-equivariant variational autoencoder that models natural illumination on the sphere without relying on 2D projections. To preserve the SO(2)-equivariance of environment maps, we use a novel Vector Neuron Vision Transformer (VN-ViT) as encoder and a rotation-equivariant conditional neural field as decoder. In the encoder, we reduce the equivariance from SO(3) to SO(2) using a novel SO(2)-equivariant fully connected layer, an extension of Vector Neurons. We show that our SO(2)-equivariant fully connected layer outperforms standard Vector Neurons when used in our SO(2)-equivariant model. Compared to previous methods, our variational autoencoder enables smoother interpolation in latent space and offers a more well-behaved latent space.