# Detecting Unexpected Obstacles for Self-Driving Cars: Fusing Deep Learning and Geometric Modeling

arXiv: https://arxiv.org/abs/1612.06573 · HTML: https://mlsanity.com/paper/1612.06573/

## Key facts

- Cited by 28 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 20 Dec 2016.

## Built on by (most cited first, 28 of 28)

- [This Looks Like Those: Illuminating Prototypical Concepts Using Multiple Visualizations](https://mlsanity.com/paper/2310.18589/) (2023)
- [Corner Cases for Visual Perception in Automated Driving: Some Guidance on Detection Approaches](https://mlsanity.com/paper/2102.05897/) (2021)
- [On the statistical complexity of quantum circuits](https://mlsanity.com/paper/2101.06154/) (2021)
- [Interpretable Image Classification with Adaptive Prototype-based Vision Transformers](https://mlsanity.com/paper/2410.20722/) (2024)
- [Unknown Sniffer for Object Detection: Don't Turn a Blind Eye to Unknown Objects](https://mlsanity.com/paper/2303.13769/) (2023)
- [Surrogate Model-Based Explainability Methods for Point Cloud NNs](https://mlsanity.com/paper/2107.13459/) (2021)
- [Recent Advances and Applications of Machine Learning in Experimental Solid Mechanics: A Review](https://mlsanity.com/paper/2303.07647/) (2023)
- [Tiny Obstacle Discovery by Occlusion-Aware Multilayer Regression](https://mlsanity.com/paper/2111.09204/) (2021)
- [Indoor Obstacle Discovery on Reflective Ground via Monocular Camera](https://mlsanity.com/paper/2401.01445/) (2024)
- [Polarization Wavefront Lidar: Learning Large Scene Reconstruction from Polarized Wavefronts](https://mlsanity.com/paper/2406.03461/) (2024)
- [TrackletMapper: Ground Surface Segmentation and Mapping from Traffic Participant Trajectories](https://mlsanity.com/paper/2209.05247/) (2022)
- [A chaotic maps-based privacy-preserving distributed deep learning for incomplete and Non-IID datasets](https://mlsanity.com/paper/2402.10145/) (2024)
- [Using a Distance Sensor to Detect Deviations in a Planar Surface](https://mlsanity.com/paper/2408.03838/) (2024)
- [Perspective Aware Road Obstacle Detection](https://mlsanity.com/paper/2210.01779/) (2022)
- [Reconstructing Humpty Dumpty: Multi-feature Graph Autoencoder for Open Set Action Recognition](https://mlsanity.com/paper/2212.06023/) (2022)
- [Fisheye Camera and Ultrasonic Sensor Fusion For Near-Field Obstacle Perception in Bird's-Eye-View](https://mlsanity.com/paper/2402.00637/) (2024)
- [End-To-End Training and Testing Gamification Framework to Learn Human Highway Driving](https://mlsanity.com/paper/2404.10849/) (2024)
- [CrossedWires: A Dataset of Syntactically Equivalent but Semantically Disparate Deep Learning Models](https://mlsanity.com/paper/2108.12768/) (2021)
- [Characterization of Semantic Segmentation Models on Mobile Platforms for Self-Navigation in Disaster-Struck Zones](https://mlsanity.com/paper/2202.01421/) (2022)
- [HazardNet: Road Debris Detection by Augmentation of Synthetic Models](https://mlsanity.com/paper/2303.07547/) (2023)
- [On the link between generative semi-supervised learning and generative open-set recognition](https://mlsanity.com/paper/2303.11702/) (2023)
- [A survey on deep learning approaches for data integration in autonomous driving system](https://mlsanity.com/paper/2306.11740/) (2023)
- [ZO-DARTS++: An Efficient and Size-Variable Zeroth-Order Neural Architecture Search Algorithm](https://mlsanity.com/paper/2503.06092/) (2025)
- [Evaluation of Multimodal Semantic Segmentation using RGB-D Data](https://mlsanity.com/paper/2103.16758/) (2021)
- [ProtoVQA: An Adaptable Prototypical Framework for Explainable Fine-Grained Visual Question Answering](https://mlsanity.com/paper/2509.16680/) (2025)
- [AVOID: The Adverse Visual Conditions Dataset with Obstacles for Driving Scene Understanding](https://mlsanity.com/paper/2512.23215/) (2025)
- [OOVDet: Low-Density Prior Learning for Zero-Shot Out-of-Vocabulary Object Detection](https://mlsanity.com/paper/2601.22685/) (2026)
- [Multi-model approach for autonomous driving: A comprehensive study on traffic sign-, vehicle- and lane detection and behavioral cloning](https://mlsanity.com/paper/2603.09255/) (2026)

## Abstract (arXiv)

The detection of small road hazards, such as lost cargo, is a vital capability for self-driving cars. We tackle this challenging and rarely addressed problem with a vision system that leverages appearance, contextual as well as geometric cues. To utilize the appearance and contextual cues, we propose a new deep learning-based obstacle detection framework. Here a variant of a fully convolutional network is used to predict a pixel-wise semantic labeling of (i) free-space, (ii) on-road unexpected obstacles, and (iii) background. The geometric cues are exploited using a state-of-the-art detection approach that predicts obstacles from stereo input images via model-based statistical hypothesis tests. We present a principled Bayesian framework to fuse the semantic and stereo-based detection results. The mid-level Stixel representation is used to describe obstacles in a flexible, compact and robust manner. We evaluate our new obstacle detection system on the Lost and Found dataset, which includes very challenging scenes with obstacles of only 5 cm height. Overall, we report a major improvement over the state-of-the-art, with relative performance gains of up to 50%. In particular, we achieve a detection rate of over 90% for distances of up to 50 m. Our system operates at 22 Hz on our self-driving platform.