mlsanityData to 7 Oct 2026

Advancing Self-supervised Monocular Depth Learning with Sparse LiDAR

arXiv 2109.09628 ↗First posted 20 Sep 2021cs.CV cs.LGcited by 13

Not on the map. Why?

Abstract (arXiv)

Self-supervised monocular depth prediction provides a cost-effective solution to obtain the 3D location of each pixel. However, the existing approaches usually lead to unsatisfactory accuracy, which is critical for autonomous robots. In this paper, we propose FusionDepth, a novel two-stage network to advance the self-supervised monocular dense depth learning by leveraging low-cost sparse (e.g. 4-beam) LiDAR. Unlike the existing methods that use sparse LiDAR mainly in a manner of time-consuming iterative post-processing, our model fuses monocular image features and sparse LiDAR features to predict initial depth maps. Then, an efficient feed-forward refine network is further designed to correct the errors in these initial depth maps in pseudo-3D space with real-time performance. Extensive experiments show that our proposed model significantly outperforms all the state-of-the-art self-supervised methods, as well as the sparse-LiDAR-based methods on both self-supervised monocular depth prediction and completion tasks. With the accurate dense depth prediction, our model outperforms the state-of-the-art sparse-LiDAR-based method (Pseudo-LiDAR++) by more than 68% for the downstream task monocular 3D object detection on the KITTI Leaderboard. Code is available at https://github.com/AutoAILab/FusionDepth

Key facts. Cited by 13 papers in the map (data through 7 Oct 2026). Builds on 73 of its 83 references that are in the map. First posted on arXiv 20 Sep 2021.

Builds on

73 of 83 references.

References · 73

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  3. 2015Rethinking the Inception Architecture for Computer Visioncited by 5,451
  4. 2014Depth Map Prediction from a Single Image using a Multi-Scale Deep Networkcited by 1,264
  5. 2019Objects as Pointscited by 1,172
  6. 2016Unsupervised Monocular Depth Estimation with Left-Right Consistencycited by 819
  7. 2017Unsupervised Learning of Depth and Ego-Motion from Videocited by 778
  8. 2018Deep Ordinal Regression Network for Monocular Depth Estimationcited by 721
  9. 2014Predicting Depth, Surface Normals and Semantic Labels with a Common Multi-Scale Convolutional Architecturecited by 483
  10. 2020Tracking Objects as Pointscited by 483
  11. 2018Pseudo-LiDAR from Visual Depth Estimation: Bridging the Gap in 3D Object Detection for Autonomous Drivingcited by 455
  12. 2019From Big to Small: Multi-Scale Local Planar Guidance for Monocular Depth Estimationcited by 379
38 more
  1. 2018GeoNet: Unsupervised Learning of Dense Depth, Optical Flow and Camera Posecited by 341
  2. 2017Sparsity Invariant CNNscited by 325
  3. 2017Joint 3D Proposal Generation and Object Detection from View Aggregationcited by 325
  4. 2019M3D-RPN: Monocular 3D Region Proposal Network for Object Detectioncited by 251
  5. 2018Competitive Collaboration: Joint Unsupervised Learning of Depth, Camera Motion, Optical Flow and Motion Segmentationcited by 222
  6. 2019Disentangling Monocular 3D Object Detectioncited by 215
  7. 2018Orthographic Feature Transform for Monocular 3D Object Detectioncited by 201
  8. 2020Non-Local Spatial Propagation Network for Depth Completioncited by 194
  9. 2019Pseudo-LiDAR++: Accurate Depth for 3D Object Detection in Autonomous Drivingcited by 188
  10. 2018Unsupervised Learning of Depth and Ego-Motion from Monocular Video Using 3D Geometric Constraintscited by 179
  11. 2019Depth from Videos in the Wild: Unsupervised Monocular Depth Learning from Unknown Camerascited by 170
  12. 2017Sparse-to-Dense: Depth Prediction from Sparse Depth Samples and a Single Imagecited by 159
  13. 2018Unsupervised Learning of Monocular Depth Estimation and Visual Odometry with Deep Feature Reconstructioncited by 159
  14. 2018Depth Prediction Without the Sensors: Leveraging Structure for Unsupervised Learning from Monocular Videoscited by 155
  15. 2017Learning Depth from Monocular Videos using Direct Methodscited by 154
  16. 2020MonoPair: Monocular 3D Object Detection Using Pairwise Spatial Relationshipscited by 153
  17. 2020RTM3D: Real-time Monocular 3D Detection from Object Keypoints for Autonomous Drivingcited by 150
  18. 2018DF-Net: Unsupervised Joint Learning of Depth and Flow using Cross-Task Consistencycited by 148
  19. 2018Self-supervised Sparse-to-Dense: Self-supervised Depth Completion from LiDAR and Monocular Cameracited by 140
  20. 2020Feature-metric Loss for Self-supervised Learning of Depth and Egomotioncited by 140
  21. 2019Accurate Monocular Object Detection via Color-Embedded 3D Reconstruction for Autonomous Drivingcited by 137
  22. 2019Self-supervised Learning with Geometric Constraints in Monocular Video: Connecting Flow, Depth, and Cameracited by 130
  23. 2018DeepLiDAR: Deep Surface Normal Guided Depth Prediction for Outdoor Scene from Sparse LiDAR Data and Single Color Imagecited by 129
  24. 2019Self-Supervised Monocular Depth Hintscited by 127
  25. 2018ROI-10D: Monocular Lifting of 2D Detection to 6D Pose and Metric Shapecited by 123
  26. 2019Monocular 3D Object Detection with Pseudo-LiDAR Point Cloudcited by 113
  27. 2019CSPN++: Learning Context and Resource Aware Convolutional Spatial Propagation Networks for Depth Completioncited by 110
  28. 2017Semi-Supervised Deep Learning for Monocular Depth Map Predictioncited by 109
  29. 2020Rethinking Pseudo-LiDAR Representationcited by 109
  30. 2019Depth Completion from Sparse LiDAR Data with Depth-Normal Constraintscited by 108
  31. 2019GS3D: An Efficient 3D Object Detection Framework for Autonomous Drivingcited by 106
  32. 2018Every Pixel Counts ++: Joint Learning of Geometry and Motion with 3D Holistic Understandingcited by 101
  33. 2017UnDeepVO: Monocular Visual Odometry through Unsupervised Deep Learningcited by 95
  34. 2019Unsupervised Depth Completion from Visual Inertial Odometrycited by 88
  35. 2018MonoGRNet: A Geometric Reasoning Network for Monocular 3D Object Localizationcited by 84
  36. 2020End-to-End Pseudo-LiDAR for Image-Based 3D Object Detectioncited by 80
  37. 2018Structured Attention Guided Convolutional Neural Fields for Monocular Depth Estimationcited by 78
  38. 2020Learning Joint 2D-3D Representations for Depth Completioncited by 71

50 most cited of 73 All 73 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 1; 2022: 2; 2023: 4; 2024: 4; 2025: 1; 2026: 1.

13 papers.

Sort:

  1. 2023Self-Supervised Multimodal Learning: A Surveycited by 58
  2. 2021CamLiFlow: Bidirectional Camera-LiDAR Fusion for Joint Optical Flow and Scene Flow Estimationcited by 56
  3. 2022Deep Depth Completion from Extremely Sparse Data: A Surveycited by 33
  4. 2024NARUTO: Neural Active Reconstruction from Uncertain Target Observationscited by 27
  5. 2023Bridging the Domain Gap: Self-Supervised 3D Scene Understanding with Foundation Modelscited by 26
  6. 2023CVRecon: Rethinking 3D Geometric Feature Learning For Neural Reconstructioncited by 16
  7. 2023Learning Optical Flow and Scene Flow with Bidirectional Camera-LiDAR Fusioncited by 10
  8. 2022Revisiting PatchMatch Multi-View Stereo for Urban 3D Reconstructioncited by 3
  9. 2024SAM-Guided Masked Token Prediction for 3D Scene Understandingcited by 3
  10. 2024DyConfidMatch: Dynamic Thresholding and Re-sampling for 3D Semi-supervised Learningcited by 1
  11. 2024SelfReDepth: Self-Supervised Real-Time Depth Restoration for Consumer-Grade Sensorscited by 0
  12. 2025Training Self-Supervised Depth Completion Using Sparse Measurements and a Single Imagecited by 0
1 more
  1. 2026DST-Calib: A Dual-Path, Self-Supervised, Target-Free LiDAR-Camera Extrinsic Calibration Networkcited by 0