# Simulating Dual-Pixel Images From Ray Tracing For Depth Estimation

arXiv: https://arxiv.org/abs/2503.11213 · HTML: https://mlsanity.com/paper/2503.11213/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 25 of its 45 references that are in the map.
- First posted on arXiv 14 Mar 2025.

## Builds on (most cited first, 25 of 25)

- [SGDR: Stochastic Gradient Descent with Warm Restarts](https://mlsanity.com/paper/1608.03983/) (2016)
- [A Large Dataset to Train Convolutional Networks for Disparity, Optical Flow, and Scene Flow Estimation](https://mlsanity.com/paper/1512.02134/) (2015)
- [Hierarchical Neural Architecture Search for Deep Stereo Matching](https://mlsanity.com/paper/2010.13501/) (2020)
- [Revisiting Single Image Depth Estimation: Toward Higher Resolution Maps with Accurate Object Boundaries](https://mlsanity.com/paper/1803.08673/) (2018)
- [Defocus Deblurring Using Dual-Pixel Data](https://mlsanity.com/paper/2005.00305/) (2020)
- [Synthetic Depth-of-Field with a Single-Camera Mobile Phone](https://mlsanity.com/paper/1806.04171/) (2018)
- [Learning Single Camera Depth Estimation using Dual-Pixels](https://mlsanity.com/paper/1904.05822/) (2019)
- [Learning to Reduce Defocus Blur by Realistically Modeling Dual-Pixel Data](https://mlsanity.com/paper/2012.03255/) (2020)
- [Dual Pixel Exploration: Simultaneous Depth Estimation and Image Restoration](https://mlsanity.com/paper/2012.00301/) (2020)
- [Curriculum Learning for ab initio Deep Learned Refractive Optics](https://mlsanity.com/paper/2302.01089/) (2023)
- [Defocus Map Estimation and Deblurring from a Single Dual-Pixel Image](https://mlsanity.com/paper/2110.05655/) (2021)
- [Du2Net: Learning Depth Estimation from Dual-Cameras and Dual-Pixels](https://mlsanity.com/paper/2003.14299/) (2020)
- [Improving Single-Image Defocus Deblurring: How Dual-Pixel Images Help Through Multi-Task Learning](https://mlsanity.com/paper/2108.05251/) (2021)
- [The Differentiable Lens: Compound Lens Search over Glass Surfaces and Materials for Object Detection](https://mlsanity.com/paper/2212.04441/) (2022)
- [Aberration-Aware Depth-from-Focus](https://mlsanity.com/paper/2303.04654/) (2023)
- [DC2: Dual-Camera Defocus Control by Learning to Refocus](https://mlsanity.com/paper/2304.03285/) (2023)
- [Facial Depth and Normal Estimation using Single Dual-Pixel Camera](https://mlsanity.com/paper/2111.12928/) (2021)
- [LDP: Language-driven Dual-Pixel Image Defocus Deblurring Network](https://mlsanity.com/paper/2307.09815/) (2023)
- [Dual-Pixel Raindrop Removal](https://mlsanity.com/paper/2210.13321/) (2022)
- [Passive Snapshot Coded Aperture Dual-Pixel RGB-D Imaging](https://mlsanity.com/paper/2402.18102/) (2024)
- [Learning Dual-Pixel Alignment for Defocus Deblurring](https://mlsanity.com/paper/2204.12105/) (2022)
- [Continuous Cost Aggregation for Dual-Pixel Disparity Extraction](https://mlsanity.com/paper/2306.07921/) (2023)
- [DMTNet: Dynamic Multi-scale Network for Dual-pixel Images Defocus Deblurring with Transformer](https://mlsanity.com/paper/2209.06040/) (2022)
- [Disparity Estimation Using a Quad-Pixel Sensor](https://mlsanity.com/paper/2409.00665/) (2024)
- [Adjust Your Focus: Defocus Deblurring From Dual-Pixel Images Using Explicit Multi-Scale Cross-Correlation](https://mlsanity.com/paper/2502.11002/) (2025)

## Built on by (most cited first, 1 of 1)

- [FoundDP: Revisiting Weak Disparity Observability in Dual-Pixel Depth Estimation](https://mlsanity.com/paper/2607.01900/) (2026)

## Abstract (arXiv)

Many studies utilize dual-pixel (DP) sensor phase characteristics for various applications, such as depth estimation and deblurring. However, since the DP image features are entirely determined by the camera hardware, DP-depth paired datasets are very scarce, especially when performing depth estimation on customized cameras. To overcome this, studies simulate DP images using ideal optical system models. However, these simulations often violate real optical propagation laws, leading to poor generalization to real DP data. To address this, we investigate the domain gap between simulated and real DP data, and propose solutions using the Simulating DP images from ray tracing (Sdirt) scheme. The Sdirt generates realistic DP images via ray tracing and integrates them into the depth estimation training pipeline. Experimental results show that models trained with Sdirt-simulated images generalize better to real DP data. The code and collected datasets will be available at github.com/LinYark/Sdirt