# Adaptive Fusion of Single-View and Multi-View Depth for Autonomous Driving

arXiv: https://arxiv.org/abs/2403.07535 · HTML: https://mlsanity.com/paper/2403.07535/

## Key facts

- Cited by 30 papers in the map (data through 14 Oct 2026).
- Builds on 43 of its 53 references that are in the map.
- First posted on arXiv 12 Mar 2024.

## Builds on (most cited first, 30 of 43)

- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [MobileNetV2: Inverted Residuals and Linear Bottlenecks](https://mlsanity.com/paper/1801.04381/) (2018)
- [A ConvNet for the 2020s](https://mlsanity.com/paper/2201.03545/) (2022)
- [ScanNet: Richly-annotated 3D Reconstructions of Indoor Scenes](https://mlsanity.com/paper/1702.04405/) (2017)
- [Depth Map Prediction from a Single Image using a Multi-Scale Deep Network](https://mlsanity.com/paper/1406.2283/) (2014)
- [ORB-SLAM2: an Open-Source SLAM System for Monocular, Stereo and RGB-D Cameras](https://mlsanity.com/paper/1610.06475/) (2016)
- [MVSNet: Depth Inference for Unstructured Multi-view Stereo](https://mlsanity.com/paper/1804.02505/) (2018)
- [Deep Ordinal Regression Network for Monocular Depth Estimation](https://mlsanity.com/paper/1806.02446/) (2018)
- [AdaBins: Depth Estimation using Adaptive Bins](https://mlsanity.com/paper/2011.14141/) (2020)
- [Cascade Cost Volume for High-Resolution Multi-View Stereo and Stereo Matching](https://mlsanity.com/paper/1912.06378/) (2019)
- [Deeper Depth Prediction with Fully Convolutional Residual Networks](https://mlsanity.com/paper/1606.00373/) (2016)
- [From Big to Small: Multi-Scale Local Planar Guidance for Monocular Depth Estimation](https://mlsanity.com/paper/1907.10326/) (2019)
- [Metric3D: Towards Zero-shot Metric 3D Prediction from A Single Image](https://mlsanity.com/paper/2307.10984/) (2023)
- [Omnidata: A Scalable Pipeline for Making Multi-Task Mid-Level Vision Datasets from 3D Scans](https://mlsanity.com/paper/2110.04994/) (2021)
- [Recurrent MVSNet for High-resolution Multi-view Stereo Depth Inference](https://mlsanity.com/paper/1902.10556/) (2019)
- [Enforcing geometric constraints of virtual normal for depth prediction](https://mlsanity.com/paper/1907.12209/) (2019)
- [PatchmatchNet: Learned Multi-View Patchmatch Stereo](https://mlsanity.com/paper/2012.01411/) (2020)
- [Super-Convergence: Very Fast Training of Neural Networks Using Large Learning Rates](https://mlsanity.com/paper/1708.07120/) (2017)
- [Iterative Geometry Encoding Volume for Stereo Matching](https://mlsanity.com/paper/2303.06615/) (2023)
- [DeepV2D: Video to Depth with Differentiable Structure from Motion](https://mlsanity.com/paper/1812.04605/) (2018)
- [Point-Based Multi-View Stereo Network](https://mlsanity.com/paper/1908.04422/) (2019)
- [Revisiting Single Image Depth Estimation: Toward Higher Resolution Maps with Accurate Object Boundaries](https://mlsanity.com/paper/1803.08673/) (2018)
- [Feature-metric Loss for Self-supervised Learning of Depth and Egomotion](https://mlsanity.com/paper/2007.10603/) (2020)
- [Attention Concatenation Volume for Accurate and Efficient Stereo Matching](https://mlsanity.com/paper/2203.02146/) (2022)
- [Fast-MVSNet: Sparse-to-Dense Multi-View Stereo With Learned Propagation and Gauss-Newton Refinement](https://mlsanity.com/paper/2003.13017/) (2020)
- [Guiding Monocular Depth Estimation Using Depth-Attention Volume](https://mlsanity.com/paper/2004.02760/) (2020)
- [Structured Attention Guided Convolutional Neural Fields for Monocular Depth Estimation](https://mlsanity.com/paper/1803.11029/) (2018)
- [Computing the Stereo Matching Cost with a Convolutional Neural Network](https://mlsanity.com/paper/1409.4326/) (2014)
- [Multi-Scale Continuous CRFs as Sequential Deep Networks for Monocular Depth Estimation](https://mlsanity.com/paper/1704.02157/) (2017)
- [DiverseDepth: Affine-invariant Depth Prediction Using Diverse Data](https://mlsanity.com/paper/2002.00569/) (2020)

## Built on by (most cited first, 30 of 30)

- [DepthSplat: Connecting Gaussian Splatting and Depth](https://mlsanity.com/paper/2410.13862/) (2024)
- [Pixel-Perfect Depth with Semantics-Prompted Diffusion Transformers](https://mlsanity.com/paper/2510.07316/) (2025)
- [Cross-View Completion Models are Zero-shot Correspondence Estimators](https://mlsanity.com/paper/2412.09072/) (2024)
- [The RoboDrive Challenge: Drive Anytime Anywhere in Any Condition](https://mlsanity.com/paper/2405.08816/) (2024)
- [MonSter++: Unified Stereo Matching, Multi-view Stereo, and Real-time Stereo with Monodepth Priors](https://mlsanity.com/paper/2501.08643/) (2025)
- [LiteVGGT: Boosting Vanilla VGGT via Geometry-aware Cached Token Merging](https://mlsanity.com/paper/2512.04939/) (2025)
- [Vision-Language Embodiment for Monocular Depth Estimation](https://mlsanity.com/paper/2503.16535/) (2025)
- [Know Your Neighbors: Improving Single-View Reconstruction via Spatial Vision-Language Reasoning](https://mlsanity.com/paper/2404.03658/) (2024)
- [PrimeDepth: Efficient Monocular Depth Estimation with a Stable Diffusion Preimage](https://mlsanity.com/paper/2409.09144/) (2024)
- [BANet: Bilateral Aggregation Network for Mobile Stereo Matching](https://mlsanity.com/paper/2503.03259/) (2025)
- [Hybrid Cost Volume for Memory-Efficient Optical Flow](https://mlsanity.com/paper/2409.04243/) (2024)
- [PromptStereo: Zero-Shot Stereo Matching via Structure and Motion Prompts](https://mlsanity.com/paper/2603.01650/) (2026)
- [Multi-modal and Multi-scale Spatial Environment Understanding for Immersive Visual Text-to-Speech](https://mlsanity.com/paper/2412.11409/) (2024)
- [SVDC: Consistent Direct Time-of-Flight Video Depth Completion with Frequency Selective Fusion](https://mlsanity.com/paper/2503.01257/) (2025)
- [PriOr-Flow: Enhancing Primitive Panoramic Optical Flow with Orthogonal View](https://mlsanity.com/paper/2506.23897/) (2025)
- [RoMeO: Robust Metric Visual Odometry](https://mlsanity.com/paper/2412.11530/) (2024)
- [Leveraging Consistent Spatio-Temporal Correspondence for Robust Visual Odometry](https://mlsanity.com/paper/2412.16923/) (2024)
- [A Wavelet-based Stereo Matching Framework for Solving Frequency Convergence Inconsistency](https://mlsanity.com/paper/2505.18024/) (2025)
- [MAFNet:Multi-frequency Adaptive Fusion Network for Real-time Stereo Matching](https://mlsanity.com/paper/2512.04358/) (2025)
- [Pixel-Perfect Visual Geometry Estimation](https://mlsanity.com/paper/2601.05246/) (2026)
- [SoPE: Spherical Coordinate-Based Positional Embedding for Enhancing Spatial Perception of 3D LVLMs](https://mlsanity.com/paper/2602.22716/) (2026)
- [RGB-Thermal Infrared Fusion for Robust Depth Estimation in Complex Environments](https://mlsanity.com/paper/2503.04821/) (2025)
- [Fairness-Aware Multi-view Evidential Learning with Adaptive Prior](https://mlsanity.com/paper/2508.12997/) (2025)
- [SynthDrive: Scalable Real2Sim2Real Sensor Simulation Pipeline for High-Fidelity Asset Generation and Driving Data Synthesis](https://mlsanity.com/paper/2509.06798/) (2025)
- [FUME: Fused Unified Multi-Gas Emission Network for Livestock Rumen Acidosis Detection](https://mlsanity.com/paper/2601.08205/) (2026)
- [StereoVGGT: A Training-Free Visual Geometry Transformer for Stereo Vision](https://mlsanity.com/paper/2603.29368/) (2026)
- [DepthVanish: Optimizing Adversarial Interval Structures for Stereo-Depth-Invisible Patches](https://mlsanity.com/paper/2506.16690/) (2025)
- [Decoupling Bidirectional Geometric Representations of 4D cost volume with 2D convolution](https://mlsanity.com/paper/2509.02415/) (2025)
- [MVS-TTA: Test-Time Adaptation for Multi-View Stereo via Meta-Auxiliary Learning](https://mlsanity.com/paper/2511.18120/) (2025)
- [GemDepth: Geometry-Embedded Features for 3D-Consistent Video Depth](https://mlsanity.com/paper/2605.10525/) (2026)

## Abstract (arXiv)

Multi-view depth estimation has achieved impressive performance over various benchmarks. However, almost all current multi-view systems rely on given ideal camera poses, which are unavailable in many real-world scenarios, such as autonomous driving. In this work, we propose a new robustness benchmark to evaluate the depth estimation system under various noisy pose settings. Surprisingly, we find current multi-view depth estimation methods or single-view and multi-view fusion methods will fail when given noisy pose settings. To address this challenge, we propose a single-view and multi-view fused depth estimation system, which adaptively integrates high-confident multi-view and single-view results for both robust and accurate depth estimations. The adaptive fusion module performs fusion by dynamically selecting high-confidence regions between two branches based on a wrapping confidence map. Thus, the system tends to choose the more reliable branch when facing textureless scenes, inaccurate calibration, dynamic objects, and other degradation or challenging conditions. Our method outperforms state-of-the-art multi-view and fusion methods under robustness testing. Furthermore, we achieve state-of-the-art performance on challenging benchmarks (KITTI and DDAD) when given accurate pose estimations. Project website: https://github.com/Junda24/AFNet/.