# 3D Visual Perception for Self-Driving Cars using a Multi-Camera System: Calibration, Mapping, Localization, and Obstacle Detection

arXiv: https://arxiv.org/abs/1708.09839 · HTML: https://mlsanity.com/paper/1708.09839/

## Key facts

- Cited by 35 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 31 Aug 2017.

## Built on by (most cited first, 30 of 35)

- [GSV-Cities: Toward Appropriate Supervised Visual Place Recognition](https://mlsanity.com/paper/2210.10239/) (2022)
- [RigNet: Repetitive Image Guided Network for Depth Completion](https://mlsanity.com/paper/2107.13802/) (2021)
- [Depth Anything with Any Prior](https://mlsanity.com/paper/2505.10565/) (2025)
- [OMNI-DC: Highly Robust Depth Completion with Multiresolution Depth Integration](https://mlsanity.com/paper/2411.19278/) (2024)
- [TransLoc3D : Point Cloud based Large-scale Place Recognition using Adaptive Receptive Fields](https://mlsanity.com/paper/2105.11605/) (2021)
- [OGNI-DC: Robust Depth Completion with Optimization-Guided Neural Iterations](https://mlsanity.com/paper/2406.11711/) (2024)
- [CASSPR: Cross Attention Single Scan Place Recognition](https://mlsanity.com/paper/2211.12542/) (2022)
- [Affine Correspondences between Multi-Camera Systems for Relative Pose Estimation](https://mlsanity.com/paper/2306.12996/) (2023)
- [PC2-PU: Patch Correlation and Point Correlation for Effective Point Cloud Upsampling](https://mlsanity.com/paper/2109.09337/) (2021)
- [RigNet++: Semantic Assisted Repetitive Image Guided Network for Depth Completion](https://mlsanity.com/paper/2309.00655/) (2023)
- [Cross-view Geo-localization with Evolving Transformer](https://mlsanity.com/paper/2107.00842/) (2021)
- [SphereVLAD++: Attention-based and Signal-enhanced Viewpoint Invariant Descriptor](https://mlsanity.com/paper/2207.02958/) (2022)
- [RGB Guided ToF Imaging System: A Survey of Deep Learning-based Methods](https://mlsanity.com/paper/2405.10357/) (2024)
- [Autonomous Driving Small-Scale Cars: A Survey of Recent Development](https://mlsanity.com/paper/2404.06229/) (2024)
- [Cross-View Image Set Geo-Localization](https://mlsanity.com/paper/2412.18852/) (2024)
- [MRGeo: Robust Cross-View Geo-Localization of Corrupted Images via Spatial and Channel Feature Enhancement](https://mlsanity.com/paper/2603.12587/) (2026)
- [DeepWORD: A GCN-based Approach for Owner-Member Relationship Detection in Autonomous Driving](https://mlsanity.com/paper/2103.16099/) (2021)
- [ClusVPR: Efficient Visual Place Recognition with Clustering-based Weighted Transformer](https://mlsanity.com/paper/2310.04099/) (2023)
- [Multicam-SLAM: Non-overlapping Multi-camera SLAM for Indirect Visual Localization and Navigation](https://mlsanity.com/paper/2406.06374/) (2024)
- [Cross-View Geo-Localization with Street-View and VHR Satellite Imagery in Decentrality Settings](https://mlsanity.com/paper/2412.11529/) (2024)
- [MOGeo: Beyond One-to-One Cross-View Object Geo-localization](https://mlsanity.com/paper/2603.13843/) (2026)
- [Constrained Bundle Adjustment for Structure From Motion Using Uncalibrated Multi-Camera Systems](https://mlsanity.com/paper/2204.04145/) (2022)
- [Practical solutions to the relative pose of three calibrated cameras](https://mlsanity.com/paper/2303.16078/) (2023)
- [Efficient Solution of Point-Line Absolute Pose](https://mlsanity.com/paper/2404.16552/) (2024)
- [Anchor-free Cross-view Object Geo-localization with Gaussian Position Encoding and Cross-view Association](https://mlsanity.com/paper/2509.25623/) (2025)
- [Seeing the Unseen: Mask-Driven Positional Encoding and Strip-Convolution Context Modeling for Cross-View Object Geo-Localization](https://mlsanity.com/paper/2510.20247/) (2025)
- [Improving Cross-view Object Geo-localization: A Dual Attention Approach with Cross-view Interaction and Multi-Scale Spatial Features](https://mlsanity.com/paper/2510.27139/) (2025)
- [Detecting Owner-member Relationship with Graph Convolution Network in Fisheye Camera System](https://mlsanity.com/paper/2201.12099/) (2022)
- [Designing an Improved Deep Learning-based Model for COVID-19 Recognition in Chest X-ray Images: A Knowledge Distillation Approach](https://mlsanity.com/paper/2301.02735/) (2023)
- [HVOFusion: Incremental Mesh Reconstruction Using Hybrid Voxel Octree](https://mlsanity.com/paper/2404.17974/) (2024)

## Abstract (arXiv)

Cameras are a crucial exteroceptive sensor for self-driving cars as they are low-cost and small, provide appearance information about the environment, and work in various weather conditions. They can be used for multiple purposes such as visual navigation and obstacle detection. We can use a surround multi-camera system to cover the full 360-degree field-of-view around the car. In this way, we avoid blind spots which can otherwise lead to accidents. To minimize the number of cameras needed for surround perception, we utilize fisheye cameras. Consequently, standard vision pipelines for 3D mapping, visual localization, obstacle detection, etc. need to be adapted to take full advantage of the availability of multiple cameras rather than treat each camera individually. In addition, processing of fisheye images has to be supported. In this paper, we describe the camera calibration and subsequent processing pipeline for multi-fisheye-camera systems developed as part of the V-Charge project. This project seeks to enable automated valet parking for self-driving cars. Our pipeline is able to precisely calibrate multi-camera systems, build sparse 3D maps for visual navigation, visually localize the car with respect to these maps, generate accurate dense maps, as well as detect obstacles based on real-time depth map extraction.