# DiffPhysCam: Differentiable Physics-Based Camera Simulation for Inverse Rendering and Embodied AI

arXiv: https://arxiv.org/abs/2508.08831 · HTML: https://mlsanity.com/paper/2508.08831/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 17 of its 48 references that are in the map.
- First posted on arXiv 12 Aug 2025.

## Builds on (most cited first, 17 of 17)

- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [The Unreasonable Effectiveness of Deep Features as a Perceptual Metric](https://mlsanity.com/paper/1801.03924/) (2018)
- [3D Gaussian Splatting for Real-Time Radiance Field Rendering](https://mlsanity.com/paper/2308.04079/) (2023)
- [Instant Neural Graphics Primitives with a Multiresolution Hash Encoding](https://mlsanity.com/paper/2201.05989/) (2022)
- [SuGaR: Surface-Aligned Gaussian Splatting for Efficient 3D Mesh Reconstruction and High-Quality Mesh Rendering](https://mlsanity.com/paper/2311.12775/) (2023)
- [Nerfstudio: A Modular Framework for Neural Radiance Field Development](https://mlsanity.com/paper/2302.04264/) (2023)
- [NeRD: Neural Reflectance Decomposition from Image Collections](https://mlsanity.com/paper/2012.03918/) (2020)
- [State of the Art on Neural Rendering](https://mlsanity.com/paper/2004.03805/) (2020)
- [Shape, Light, and Material Decomposition from Images using Monte Carlo Rendering and Denoising](https://mlsanity.com/paper/2206.03380/) (2022)
- [VISTA 2.0: An Open, Data-driven Simulator for Multimodal Sensing and Policy Learning for Autonomous Vehicles](https://mlsanity.com/paper/2111.12083/) (2021)
- [Neural Network Generalization: The impact of camera parameters](https://mlsanity.com/paper/1912.03604/) (2019)
- [Modeling Camera Effects to Improve Visual Learning from Synthetic Data](https://mlsanity.com/paper/1803.07721/) (2018)
- [Neural Camera Simulators](https://mlsanity.com/paper/2104.05237/) (2021)
- [Sensor Transfer: Learning Optimal Sensor Effect Image Augmentation for Sim-to-Real Domain Adaptation](https://mlsanity.com/paper/1809.06256/) (2018)
- [NeRFocus: Neural Radiance Field for 3D Synthetic Defocus](https://mlsanity.com/paper/2203.05189/) (2022)
- [DriveEnv-NeRF: Exploration of A NeRF-Based Autonomous Driving Environment for Real-World Performance Validation](https://mlsanity.com/paper/2403.15791/) (2024)
- [Learning autonomous driving from aerial imagery](https://mlsanity.com/paper/2410.14177/) (2024)

## Abstract (arXiv)

Generating synthetic images that closely mimic those from real cameras is instrumental in training visual models and enabling end-to-end visuomotor learning. We introduce DiffPhysCam, a differentiable camera simulator designed to support robotics and embodied AI applications by enabling gradient-based optimization in visual perception pipelines. Differentiable rendering also allows inverse reconstruction of real-world scenes as digital twins, facilitating simulation-based robotics training. Existing virtual cameras offer limited control over intrinsic settings, have difficulty capturing optical artifacts, and lack tunable calibration parameters. DiffPhysCam addresses these limitations through a multi-stage pipeline that provides fine-grained control over camera settings, models key optical effects such as defocus blur, and supports calibration with real-world data. It enables both forward rendering for image synthesis and inverse rendering for 3D scene reconstruction, including mesh and material texture optimization. We show that DiffPhysCam enhances robotic perception performance in synthetic image tasks. As an illustrative example, we create a digital twin of a real-world scene using inverse rendering and use it to set up a virtual experiment in a multi-physics simulation, in which we demonstrate navigation of an autonomous ground vehicle using images generated by DiffPhysCam. The code, data, and output results associated with this paper are available online for reproducibility at Chen, Bo-Hsun (2026a,b,c).