# Hybrelighter: Combining Deep Anisotropic Diffusion and Scene Reconstruction for On-device Real-time Relighting in Mixed Reality

arXiv: https://arxiv.org/abs/2508.14930 · HTML: https://mlsanity.com/paper/2508.14930/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 23 of its 28 references that are in the map.
- First posted on arXiv 19 Aug 2025.

## Builds on (most cited first, 23 of 23)

- [The Unreasonable Effectiveness of Deep Features as a Perceptual Metric](https://mlsanity.com/paper/1801.03924/) (2018)
- [ARKitScenes: A Diverse Real-World Dataset For 3D Indoor Scene Understanding Using Mobile RGB-D Data](https://mlsanity.com/paper/2111.08897/) (2021)
- [Metric3Dv2: A Versatile Monocular Geometric Foundation Model for Zero-shot Metric Depth and Surface Normal Estimation](https://mlsanity.com/paper/2404.15506/) (2024)
- [Video Depth Anything: Consistent Depth Estimation for Super-Long Videos](https://mlsanity.com/paper/2501.12375/) (2025)
- [CityGaussian: Real-time High-quality Large-Scale Scene Rendering with Gaussians](https://mlsanity.com/paper/2404.01133/) (2024)
- [MeshLRM: Large Reconstruction Model for High-Quality Meshes](https://mlsanity.com/paper/2404.12385/) (2024)
- [SLAM3R: Real-Time Dense Scene Reconstruction from Monocular RGB Videos](https://mlsanity.com/paper/2412.09401/) (2024)
- [Neural Gaffer: Relighting Any Object via Diffusion](https://mlsanity.com/paper/2406.07520/) (2024)
- [SwitchLight: Co-design of Physics-driven Architecture and Pre-training Framework for Human Portrait Relighting](https://mlsanity.com/paper/2402.18848/) (2024)
- [MeshFormer: High-Quality Mesh Generation with 3D-Guided Reconstruction Model](https://mlsanity.com/paper/2408.10198/) (2024)
- [Guided Depth Super-Resolution by Deep Anisotropic Diffusion](https://mlsanity.com/paper/2211.11592/) (2022)
- [GS^3: Efficient Relighting with Triple Gaussian Splatting](https://mlsanity.com/paper/2410.11419/) (2024)
- [Relighting Neural Radiance Fields with Shadow and Highlight Hints](https://mlsanity.com/paper/2308.13404/) (2023)
- [Spherical Space Feature Decomposition for Guided Depth Map Super-Resolution](https://mlsanity.com/paper/2303.08942/) (2023)
- [SDXS: Real-Time One-Step Latent Diffusion Models with Image Conditions](https://mlsanity.com/paper/2403.16627/) (2024)
- [DenseLiDAR: A Real-Time Pseudo Dense Depth Guided Depth Completion Network](https://mlsanity.com/paper/2108.12655/) (2021)
- [Consistent Direct Time-of-Flight Video Depth Super-Resolution](https://mlsanity.com/paper/2211.08658/) (2022)
- [Weakly-supervised Single-view Image Relighting](https://mlsanity.com/paper/2303.13852/) (2023)
- [Multi-modal Bifurcated Network for Depth Guided Image Relighting](https://mlsanity.com/paper/2105.00690/) (2021)
- [Real-time Monocular Depth Estimation on Embedded Systems](https://mlsanity.com/paper/2308.10569/) (2023)
- [DeCoTR: Enhancing Depth Completion with 2D and 3D Attentions](https://mlsanity.com/paper/2403.12202/) (2024)
- [Compressed Depth Map Super-Resolution and Restoration: AIM 2024 Challenge Results](https://mlsanity.com/paper/2409.16277/) (2024)
- [HVOFusion: Incremental Mesh Reconstruction Using Hybrid Voxel Octree](https://mlsanity.com/paper/2404.17974/) (2024)

## Built on by (most cited first, 1 of 1)

- [Decoupled Illumination Priors for Spatially Controllable Multi-View Indoor Scene Relighting](https://mlsanity.com/paper/2607.08879/) (2026)

## Abstract (arXiv)

Mixed Reality scene relighting, where virtual changes to lighting conditions realistically interact with physical objects, producing authentic illumination and shadows, can be used in a variety of applications. One such application in real estate could be visualizing a room at different times of day and placing virtual light fixtures. Existing deep learning-based relighting techniques typically exceed the real-time performance capabilities of current MR devices. On the other hand, scene understanding methods, such as on-device scene reconstruction, often yield inaccurate results due to scanning limitations, in turn affecting relighting quality. Finally, simpler 2D image filter-based approaches cannot represent complex geometry and shadows. We introduce a novel method to integrate image segmentation, with lighting propagation via anisotropic diffusion on top of basic scene understanding, and the computational simplicity of filter-based techniques. Our approach corrects on-device scanning inaccuracies, delivering visually appealing and accurate relighting effects in real-time on edge devices, achieving speeds as high as 100 fps. We show a direct comparison between our method and the industry standard, and present a practical demonstration of our method in the aforementioned real estate example.