# Close the Optical Sensing Domain Gap by Physics-Grounded Active Stereo Sensor Simulation

arXiv: https://arxiv.org/abs/2201.11924 · HTML: https://mlsanity.com/paper/2201.11924/

## Key facts

- Cited by 16 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 28 Jan 2022.

## Built on by (most cited first, 16 of 16)

- [RoboGen: Towards Unleashing Infinite Data for Automated Robot Learning via Generative Simulation](https://mlsanity.com/paper/2311.01455/) (2023)
- [Flow as the Cross-Domain Manipulation Interface](https://mlsanity.com/paper/2407.15208/) (2024)
- [Pure Vision Language Action (VLA) Models: A Comprehensive Survey](https://mlsanity.com/paper/2509.19012/) (2025)
- [Diffusion Dynamics Models with Generative State Estimation for Cloth Manipulation](https://mlsanity.com/paper/2503.11999/) (2025)
- [CenterGrasp: Object-Aware Implicit Representation Learning for Simultaneous Shape Reconstruction and 6-DoF Grasp Estimation](https://mlsanity.com/paper/2312.08240/) (2023)
- [Sim2Real Manipulation on Unknown Objects with Tactile-based Reinforcement Learning](https://mlsanity.com/paper/2403.12170/) (2024)
- [Towards Generalist Robots: A Promising Paradigm via Generative Simulation](https://mlsanity.com/paper/2305.10455/) (2023)
- [Part-Guided 3D RL for Sim2Real Articulated Object Manipulation](https://mlsanity.com/paper/2404.17302/) (2024)
- [RFTrans: Leveraging Refractive Flow of Transparent Objects for Surface Normal Estimation and Manipulation](https://mlsanity.com/paper/2311.12398/) (2023)
- [Enhancing Generalizable 6D Pose Tracking of an In-Hand Object with Tactile Sensing](https://mlsanity.com/paper/2210.04026/) (2022)
- [RaSim: A Range-aware High-fidelity RGB-D Data Simulation Pipeline for Real-world Applications](https://mlsanity.com/paper/2404.03962/) (2024)
- [Sim-to-Real Grasp Detection with Global-to-Local RGB-D Adaptation](https://mlsanity.com/paper/2403.11511/) (2024)
- [Aligning Perception, Reasoning, Modeling and Interaction: A Survey on Physical AI](https://mlsanity.com/paper/2510.04978/) (2025)
- [TransTouch: Learning Transparent Objects Depth Sensing Through Sparse Touches](https://mlsanity.com/paper/2309.09427/) (2023)
- [Stable-Sim2Real: Exploring Simulation of Real-Captured 3D Data with Two-Stage Depth Diffusion](https://mlsanity.com/paper/2507.23483/) (2025)
- [CenterArt: Joint Shape Reconstruction and 6-DoF Grasp Estimation of Articulated Objects](https://mlsanity.com/paper/2404.14968/) (2024)

## Abstract (arXiv)

In this paper, we focus on the simulation of active stereovision depth sensors, which are popular in both academic and industry communities. Inspired by the underlying mechanism of the sensors, we designed a fully physics-grounded simulation pipeline that includes material acquisition, ray-tracing-based infrared (IR) image rendering, IR noise simulation, and depth estimation. The pipeline is able to generate depth maps with material-dependent error patterns similar to a real depth sensor in real time. We conduct real experiments to show that perception algorithms and reinforcement learning policies trained in our simulation platform could transfer well to the real-world test cases without any fine-tuning. Furthermore, due to the high degree of realism of this simulation, our depth sensor simulator can be used as a convenient testbed to evaluate the algorithm performance in the real world, which will largely reduce the human effort in developing robotic algorithms. The entire pipeline has been integrated into the SAPIEN simulator and is open-sourced to promote the research of vision and robotics communities.