# Physically-based Lighting Generation for Robotic Manipulation

arXiv: https://arxiv.org/abs/2508.01442 · HTML: https://mlsanity.com/paper/2508.01442/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 2 Aug 2025.

## Built on by (most cited first, 5 of 5)

- [EMMA: Generalizing Real-World Robot Manipulation via Generative Visual Transfer](https://mlsanity.com/paper/2509.22407/) (2025)
- [Tri-Info: Generalizable, Interpretable Failure Prediction for VLA Models via Information Theory](https://mlsanity.com/paper/2606.19998/) (2026)
- [RoHIL: Robust Human-in-the-Loop Robotic Reinforcement Learning Against Illumination Variations](https://mlsanity.com/paper/2605.19924/) (2026)
- [Lighting-aware Unified Model for Instance Segmentation](https://mlsanity.com/paper/2605.20436/) (2026)
- [MinInter: Minimizing Trajectory Interpolation During Data Augmentation for Imitation Learning](https://mlsanity.com/paper/2606.24078/) (2026)

## Abstract (arXiv)

In this paper, we propose the first framework that leverages physically-based inverse rendering for novel lighting generation on existing real-world human demonstrations of robotic manipulation tasks. Specifically, inverse rendering decomposes the first frame in each demonstration into geometric (surface normal, depth) and material (albedo, roughness, metallic) properties, which are then used to render appearance changes under different lighting sources. To improve efficiency and maintain consistency across each generated sequence, we fine-tune Stable Video Diffusion on robot execution videos for temporal lighting propagation. We evaluate our framework by measuring the visual quality of the generated sequences, assessing its effectiveness in improving the imitation learning policy performance (38.75%) under six unseen real-world lighting conditions, and conduct ablation studies on individual modules of the proposed framework. We further showcase three downstream applications enabled by the proposed framework: background generation, object texture generation and distractor positioning. The code for the framework will be made publicly available.