# StereoVoxelNet: Real-Time Obstacle Detection Based on Occupancy Voxels from a Stereo Camera Using Deep Neural Networks

arXiv: https://arxiv.org/abs/2209.08459 · HTML: https://mlsanity.com/paper/2209.08459/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- Builds on 28 of its 47 references that are in the map.
- First posted on arXiv 18 Sep 2022.

## Builds on (most cited first, 28 of 28)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [nuScenes: A multimodal dataset for autonomous driving](https://mlsanity.com/paper/1903.11027/) (2019)
- [Scalability in Perception for Autonomous Driving: Waymo Open Dataset](https://mlsanity.com/paper/1912.04838/) (2019)
- [End to End Learning for Self-Driving Cars](https://mlsanity.com/paper/1604.07316/) (2016)
- [A Large Dataset to Train Convolutional Networks for Disparity, Optical Flow, and Scene Flow Estimation](https://mlsanity.com/paper/1512.02134/) (2015)
- [3D Semantic Segmentation with Submanifold Sparse Convolutional Networks](https://mlsanity.com/paper/1711.10275/) (2017)
- [3D-R2N2: A Unified Approach for Single and Multi-view 3D Object Reconstruction](https://mlsanity.com/paper/1604.00449/) (2016)
- [Submanifold Sparse Convolutional Networks](https://mlsanity.com/paper/1706.01307/) (2017)
- [Octree Generating Networks: Efficient Convolutional Architectures for High-resolution 3D Outputs](https://mlsanity.com/paper/1703.09438/) (2017)
- [Hierarchical Neural Architecture Search for Deep Stereo Matching](https://mlsanity.com/paper/2010.13501/) (2020)
- [CFNet: Cascade and Fused Cost Volume for Robust Stereo Matching](https://mlsanity.com/paper/2104.04314/) (2021)
- [Stereo Matching by Training a Convolutional Neural Network to Compare Image Patches](https://mlsanity.com/paper/1510.05970/) (2015)
- [Attention Concatenation Volume for Accurate and Efficient Stereo Matching](https://mlsanity.com/paper/2203.02146/) (2022)
- [Learning High-Speed Flight in the Wild](https://mlsanity.com/paper/2110.05113/) (2021)
- [Socially Compliant Navigation Dataset (SCAND): A Large-Scale Dataset of Demonstrations for Social Navigation](https://mlsanity.com/paper/2203.15041/) (2022)
- [Monocular 3D Object Detection with Pseudo-LiDAR Point Cloud](https://mlsanity.com/paper/1903.09847/) (2019)
- [MarrNet: 3D Shape Reconstruction via 2.5D Sketches](https://mlsanity.com/paper/1711.03129/) (2017)
- [Perspective Transformer Nets: Learning Single-View 3D Object Reconstruction without 3D Supervision](https://mlsanity.com/paper/1612.00814/) (2016)
- [MobileStereoNet: Towards Lightweight Deep Networks for Stereo Matching](https://mlsanity.com/paper/2108.09770/) (2021)
- [Local Similarity Pattern and Cost Self-Reassembling for Deep Stereo Matching Networks](https://mlsanity.com/paper/2112.01011/) (2021)
- [Anytime Dense Prediction with Confidence Adaptivity](https://mlsanity.com/paper/2104.00749/) (2021)
- [Leveraging Stereo-Camera Data for Real-Time Dynamic Obstacle Detection and Tracking](https://mlsanity.com/paper/2007.10743/) (2020)
- [Quadtree Generating Networks: Efficient Hierarchical Scene Parsing with Sparse Convolutions](https://mlsanity.com/paper/1907.11821/) (2019)
- [Pushbroom Stereo for High-Speed Navigation in Cluttered Environments](https://mlsanity.com/paper/1407.7091/) (2014)
- [Deep Reinforcement Learning based Robot Navigation in Dynamic Environments using Occupancy Values of Motion Primitives](https://mlsanity.com/paper/2208.08034/) (2022)
- [Detecting and Mapping Trees in Unstructured Environments with a Stereo Camera and Pseudo-Lidar](https://mlsanity.com/paper/2103.15967/) (2021)
- [Reactive Navigation Framework for Mobile Robots by Heuristically Evaluated Pre-sampled Trajectories](https://mlsanity.com/paper/2105.08145/) (2021)

## Built on by (most cited first, 6 of 6)

- [Cam4DOcc: Benchmark for Camera-Only 4D Occupancy Forecasting in Autonomous Driving Applications](https://mlsanity.com/paper/2311.17663/) (2023)
- [ViHOPE: Visuotactile In-Hand Object 6D Pose Estimation with Shape Completion](https://mlsanity.com/paper/2309.05662/) (2023)
- [Recent Advancements in Deep Learning Applications and Methods for Autonomous Navigation: A Comprehensive Review](https://mlsanity.com/paper/2302.11089/) (2023)
- [StereoPolicy: Improving Robotic Manipulation Policies via Stereo Perception](https://mlsanity.com/paper/2605.09989/) (2026)
- [Humanoid-OmniOcc: Stereo-Based Full-View Occupancy Dataset for Embodied AI](https://mlsanity.com/paper/2606.22971/) (2026)
- [ODTFormer: Efficient Obstacle Detection and Tracking with Stereo Cameras Based on Transformer](https://mlsanity.com/paper/2403.14626/) (2024)

## Abstract (arXiv)

Obstacle detection is a safety-critical problem in robot navigation, where stereo matching is a popular vision-based approach. While deep neural networks have shown impressive results in computer vision, most of the previous obstacle detection works only leverage traditional stereo matching techniques to meet the computational constraints for real-time feedback. This paper proposes a computationally efficient method that employs a deep neural network to detect occupancy from stereo images directly. Instead of learning the point cloud correspondence from the stereo data, our approach extracts the compact obstacle distribution based on volumetric representations. In addition, we prune the computation of safety irrelevant spaces in a coarse-to-fine manner based on octrees generated by the decoder. As a result, we achieve real-time performance on the onboard computer (NVIDIA Jetson TX2). Our approach detects obstacles accurately in the range of 32 meters and achieves better IoU (Intersection over Union) and CD (Chamfer Distance) scores with only 2% of the computation cost of the state-of-the-art stereo model. Furthermore, we validate our method's robustness and real-world feasibility through autonomous navigation experiments with a real robot. Hence, our work contributes toward closing the gap between the stereo-based system in robot perception and state-of-the-art stereo models in computer vision. To counter the scarcity of high-quality real-world indoor stereo datasets, we collect a 1.36 hours stereo dataset with a mobile robot which is used to fine-tune our model. The dataset, the code, and further details including additional visualizations are available at https://lhy.xyz/stereovoxelnet