# LIME: Live Intrinsic Material Estimation

arXiv: https://arxiv.org/abs/1801.01075 · HTML: https://mlsanity.com/paper/1801.01075/

## Key facts

- Cited by 33 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 3 Jan 2018.

## Built on by (most cited first, 30 of 33)

- [NeRF for Outdoor Scene Relighting](https://mlsanity.com/paper/2112.05140/) (2021)
- [Deep Polarization Imaging for 3D shape and SVBRDF Acquisition](https://mlsanity.com/paper/2105.02875/) (2021)
- [Intrinsic Image Decomposition via Ordinal Shading](https://mlsanity.com/paper/2311.12792/) (2023)
- [Free-viewpoint Indoor Neural Relighting from Multi-view Stereo](https://mlsanity.com/paper/2106.13299/) (2021)
- [Self-supervised Outdoor Scene Relighting](https://mlsanity.com/paper/2107.03106/) (2021)
- [Colorful Diffuse Intrinsic Image Decomposition in the Wild](https://mlsanity.com/paper/2409.13690/) (2024)
- [A Survey on Intrinsic Images: Delving Deep Into Lambert and Beyond](https://mlsanity.com/paper/2112.03842/) (2021)
- [VoLux-GAN: A Generative Model for 3D Face Synthesis with HDRI Relighting](https://mlsanity.com/paper/2201.04873/) (2022)
- [Shape and Material Capture at Home](https://mlsanity.com/paper/2104.06397/) (2021)
- [PhotoScene: Photorealistic Material and Lighting Transfer for Indoor Scenes](https://mlsanity.com/paper/2207.00757/) (2022)
- [LIRM: Large Inverse Rendering Model for Progressive Reconstruction of Shape, Materials and View-dependent Radiance Fields](https://mlsanity.com/paper/2504.20026/) (2025)
- [Weakly-supervised Single-view Image Relighting](https://mlsanity.com/paper/2303.13852/) (2023)
- [Relightable Neural Human Assets from Multi-view Gradient Illuminations](https://mlsanity.com/paper/2212.07648/) (2022)
- [Controllable Light Diffusion for Portraits](https://mlsanity.com/paper/2305.04745/) (2023)
- [Diffusion Reflectance Map: Single-Image Stochastic Inverse Rendering of Illumination and Reflectance](https://mlsanity.com/paper/2312.04529/) (2023)
- [Inverse Rendering of Translucent Objects using Physical and Neural Renderers](https://mlsanity.com/paper/2305.08336/) (2023)
- [GAN2X: Non-Lambertian Inverse Rendering of Image GANs](https://mlsanity.com/paper/2206.09244/) (2022)
- [VQ-NeRF: Neural Reflectance Decomposition and Editing with Vector Quantization](https://mlsanity.com/paper/2310.11864/) (2023)
- [DeRenderNet: Intrinsic Image Decomposition of Urban Scenes with Shape-(In)dependent Shading Rendering](https://mlsanity.com/paper/2104.13602/) (2021)
- [Deep Polarization Cues for Single-shot Shape and Subsurface Scattering Estimation](https://mlsanity.com/paper/2407.08149/) (2024)
- [One-shot recognition of any material anywhere using contrastive learning with physics-based rendering](https://mlsanity.com/paper/2212.00648/) (2022)
- [Learning to Remove Wrinkled Transparent Film with Polarized Prior](https://mlsanity.com/paper/2403.04368/) (2024)
- [MERLiN: Single-Shot Material Estimation and Relighting for Photometric Stereo](https://mlsanity.com/paper/2409.00674/) (2024)
- [InvRGB+L: Inverse Rendering of Complex Scenes with Unified Color and LiDAR Reflectance Modeling](https://mlsanity.com/paper/2507.17613/) (2025)
- [Style Agnostic 3D Reconstruction via Adversarial Style Transfer](https://mlsanity.com/paper/2110.10784/) (2021)
- [DPF: Learning Dense Prediction Fields with Weak Supervision](https://mlsanity.com/paper/2303.16890/) (2023)
- [LSRM: High-Fidelity Object-Centric Reconstruction via Scaled Context Windows](https://mlsanity.com/paper/2604.05182/) (2026)
- [Intrinsic Image Transfer for Illumination Manipulation](https://mlsanity.com/paper/2107.00704/) (2021)
- [Predicting Surface Reflectance Properties of Outdoor Scenes Under Unknown Natural Illumination](https://mlsanity.com/paper/2105.06820/) (2021)
- [NeIF: Representing General Reflectance as Neural Intrinsics Fields for Uncalibrated Photometric Stereo](https://mlsanity.com/paper/2208.08897/) (2022)

## Abstract (arXiv)

We present the first end to end approach for real time material estimation for general object shapes with uniform material that only requires a single color image as input. In addition to Lambertian surface properties, our approach fully automatically computes the specular albedo, material shininess, and a foreground segmentation. We tackle this challenging and ill posed inverse rendering problem using recent advances in image to image translation techniques based on deep convolutional encoder decoder architectures. The underlying core representations of our approach are specular shading, diffuse shading and mirror images, which allow to learn the effective and accurate separation of diffuse and specular albedo. In addition, we propose a novel highly efficient perceptual rendering loss that mimics real world image formation and obtains intermediate results even during run time. The estimation of material parameters at real time frame rates enables exciting mixed reality applications, such as seamless illumination consistent integration of virtual objects into real world scenes, and virtual material cloning. We demonstrate our approach in a live setup, compare it to the state of the art, and demonstrate its effectiveness through quantitative and qualitative evaluation.