# UnrealStereo: Controlling Hazardous Factors to Analyze Stereo Vision

arXiv: https://arxiv.org/abs/1612.04647 · HTML: https://mlsanity.com/paper/1612.04647/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 14 Dec 2016.

## Built on by (most cited first, 8 of 8)

- [RTMV: A Ray-Traced Multi-View Synthetic Dataset for Novel View Synthesis](https://mlsanity.com/paper/2205.07058/) (2022)
- [NViSII: A Scriptable Tool for Photorealistic Image Generation](https://mlsanity.com/paper/2105.13962/) (2021)
- [Object Detection and Pose Estimation from RGB and Depth Data for Real-time, Adaptive Robotic Grasping](https://mlsanity.com/paper/2101.07347/) (2021)
- [Learning Part Segmentation from Synthetic Animals](https://mlsanity.com/paper/2311.18661/) (2023)
- [Improving Pallet Detection Using Synthetic Data](https://mlsanity.com/paper/2402.07098/) (2024)
- [Analyzing Effects of Fake Training Data on the Performance of Deep Learning Systems](https://mlsanity.com/paper/2303.01268/) (2023)
- [Bent & Broken Bicycles: Leveraging synthetic data for damaged object re-identification](https://mlsanity.com/paper/2304.07883/) (2023)
- [Rapid Training Data Creation by Synthesizing Medical Images for Classification and Localization](https://mlsanity.com/paper/2308.04687/) (2023)

## Abstract (arXiv)

A reliable stereo algorithm is critical for many robotics applications. But textureless and specular regions can easily cause failure by making feature matching difficult. Understanding whether an algorithm is robust to these hazardous regions is important. Although many stereo benchmarks have been developed to evaluate performance, it is hard to quantify the effect of hazardous regions in real images because the location and severity of these regions are unknown. In this paper, we develop a synthetic image generation tool enabling to control hazardous factors, such as making objects more specular or transparent, to produce hazardous regions at different degrees. The densely controlled sampling strategy in virtual worlds enables to effectively stress test stereo algorithms by varying the types and degrees of the hazard. We generate a large synthetic image dataset with automatically computed hazardous regions and analyze algorithms on these regions. The observations from synthetic images are further validated by annotating hazardous regions in real-world datasets Middlebury and KITTI (which gives a sparse sampling of the hazards). Our synthetic image generation tool is based on a game engine Unreal Engine 4 and will be open-source along with the virtual scenes in our experiments. Many publicly available realistic game contents can be used by our tool to provide an enormous resource for development and evaluation of algorithms.