mlsanityData to 7 Oct 2026

The Temporal Opportunist: Self-Supervised Multi-Frame Monocular Depth

arXiv 2104.14540 ↗First posted 29 Apr 2021cs.CVcited by 190

Not on the map. Why?

Abstract (arXiv)

Self-supervised monocular depth estimation networks are trained to predict scene depth using nearby frames as a supervision signal during training. However, for many applications, sequence information in the form of video frames is also available at test time. The vast majority of monocular networks do not make use of this extra signal, thus ignoring valuable information that could be used to improve the predicted depth. Those that do, either use computationally expensive test-time refinement techniques or off-the-shelf recurrent networks, which only indirectly make use of the geometric information that is inherently available. We propose ManyDepth, an adaptive approach to dense depth estimation that can make use of sequence information at test time, when it is available. Taking inspiration from multi-view stereo, we propose a deep end-to-end cost volume based approach that is trained using self-supervision only. We present a novel consistency loss that encourages the network to ignore the cost volume when it is deemed unreliable, e.g. in the case of moving objects, and an augmentation scheme to cope with static cameras. Our detailed experiments on both KITTI and Cityscapes show that we outperform all published self-supervised baselines, including those that use single or multiple frames at test time.

Key facts. Cited by 190 papers in the map (data through 7 Oct 2026). Builds on 76 of its 102 references that are in the map. First posted on arXiv 29 Apr 2021.

Builds on

76 of 102 references.

References · 76

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  3. 2014ImageNet Large Scale Visual Recognition Challengecited by 8,441
  4. 2016The Cityscapes Dataset for Semantic Urban Scene Understandingcited by 4,427
  5. 2014Depth Map Prediction from a Single Image using a Multi-Scale Deep Networkcited by 1,264
  6. 2015FlowNet: Learning Optical Flow with Convolutional Networkscited by 1,054
  7. 2015A Large Dataset to Train Convolutional Networks for Disparity, Optical Flow, and Scene Flow Estimationcited by 878
  8. 2016Unsupervised Monocular Depth Estimation with Left-Right Consistencycited by 819
  9. 2018MVSNet: Depth Inference for Unstructured Multi-view Stereocited by 780
  10. 2017Unsupervised Learning of Depth and Ego-Motion from Videocited by 778
  11. 2018Deep Ordinal Regression Network for Monocular Depth Estimationcited by 721
  12. 2018Pyramid Stereo Matching Networkcited by 486
38 more
  1. 2014Predicting Depth, Surface Normals and Semantic Labels with a Common Multi-Scale Convolutional Architecturecited by 483
  2. 20193D Packing for Self-Supervised Monocular Depth Estimationcited by 437
  3. 2019Cascade Cost Volume for High-Resolution Multi-View Stereo and Stereo Matchingcited by 414
  4. 2016Unsupervised CNN for Single View Depth Estimation: Geometry to the Rescuecited by 368
  5. 2017End-to-End Learning of Geometry and Context for Deep Stereo Regressioncited by 359
  6. 2018GeoNet: Unsupervised Learning of Dense Depth, Optical Flow and Camera Posecited by 341
  7. 2017Sparsity Invariant CNNscited by 325
  8. 2018DeepMVS: Learning Multi-view Stereopsiscited by 277
  9. 2019Unsupervised Scale-consistent Depth and Ego-motion Learning from Monocular Videocited by 244
  10. 2019GA-Net: Guided Aggregation Net for End-to-end Stereo Matchingcited by 242
  11. 2018Competitive Collaboration: Joint Unsupervised Learning of Depth, Camera Motion, Optical Flow and Motion Segmentationcited by 222
  12. 2020Consistent Video Depth Estimationcited by 220
  13. 2016DeMoN: Depth and Motion Network for Learning Monocular Stereocited by 200
  14. 2017Learning a Multi-View Stereo Machinecited by 194
  15. 2016Single-Image Depth Perception in the Wildcited by 179
  16. 2019Depth from Videos in the Wild: Unsupervised Monocular Depth Learning from Unknown Camerascited by 170
  17. 2018Unsupervised Learning of Monocular Depth Estimation and Visual Odometry with Deep Feature Reconstructioncited by 159
  18. 2018Depth Prediction Without the Sensors: Leveraging Structure for Unsupervised Learning from Monocular Videoscited by 155
  19. 2017CNN-SLAM: Real-time dense monocular SLAM with learned depth predictioncited by 152
  20. 2018CodeSLAM - Learning a Compact, Optimisable Representation for Dense Visual SLAMcited by 151
  21. 2020Self-Supervised Monocular Depth Estimation: Solving the Dynamic Object Problem by Semantic Guidancecited by 148
  22. 2020Feature-metric Loss for Self-supervised Learning of Depth and Egomotioncited by 140
  23. 2018Learning Depth with Convolutional Spatial Propagation Networkcited by 135
  24. 2020Semantically-Guided Representation Learning for Self-Supervised Monocular Depthcited by 134
  25. 2019Self-supervised Learning with Geometric Constraints in Monocular Video: Connecting Flow, Depth, and Cameracited by 130
  26. 2017SurfaceNet: An End-to-end 3D Neural Network for Multiview Stereopsiscited by 128
  27. 2019Self-Supervised Monocular Depth Hintscited by 127
  28. 2019Learning the Depths of Moving People by Watching Frozen Peoplecited by 120
  29. 2020Self-supervised Monocular Trained Depth Estimation using Self-attention and Discrete Disparity Volumecited by 116
  30. 2019DPSNet: End-to-end Deep Plane Sweep Stereocited by 115
  31. 2019Domain-invariant Stereo Matching Networkscited by 108
  32. 2018Every Pixel Counts ++: Joint Learning of Geometry and Motion with 3D Holistic Understandingcited by 101
  33. 2017SfM-Net: Learning of Structure and Motion from Videocited by 89
  34. 2019CAM-Convs: Camera-Aware Multi-Scale Convolutions for Single-View Depthcited by 75
  35. 2016Deep3D: Fully Automatic 2D-to-3D Video Conversion with Deep Convolutional Neural Networkscited by 72
  36. 2020Don't Forget The Past: Recurrent Depth Estimation from Monocular Videocited by 71
  37. 2019DeepSFM: Structure From Motion Via Deep Bundle Adjustmentcited by 70
  38. 2019Neural RGB->D Sensing: Depth and Uncertainty from a Video Cameracited by 69

50 most cited of 76 All 76 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 7; 2022: 36; 2023: 58; 2024: 48; 2025: 27; 2026: 14.

190 papers.

Sort:

By area:

  1. 2024MonST3R: A Simple Approach for Estimating Geometry in the Presence of Motioncited by 431
  2. 2022Unifying Flow, Stereo and Depth Estimationcited by 270
  3. 2022P3Depth: Monocular Depth Estimation with a Piecewise Planarity Priorcited by 106
  4. 2024UniScene: Unified Occupancy-centric Driving Scene Generationcited by 102
  5. 2024Learning Temporally Consistent Video Depth from Video Diffusion Priorscited by 100
  6. 2022LocalBins: Improving Depth Estimation by Learning Local Distributionscited by 92
  7. 2022SimpleRecon: 3D Reconstruction Without 3D Convolutionscited by 83
  8. 2021Self-Supervised Monocular Depth Estimation with Internal Feature Fusioncited by 67
  9. 2022Multi-Frame Self-Supervised Depth with Transformerscited by 64
  10. 2022Physical Attack on Monocular Depth Estimation with Optimal Adversarial Patchescited by 57
  11. 2022STS: Surround-view Temporal Stereo for Multi-view 3D Detectioncited by 53
  12. 2022MVSTER: Epipolar Transformer for Efficient Multi-View Stereocited by 50
38 more
  1. 2022Disentangling Object Motion and Occlusion for Unsupervised Multi-frame Monocular Depthcited by 48
  2. 2023RoboDepth: Robust Out-of-Distribution Depth Estimation under Corruptionscited by 47
  3. 2021Toward Practical Monocular Indoor Depth Estimationcited by 42
  4. 2023R3D3: Dense 3D Reconstruction of Dynamic Scenes from Multiple Camerascited by 39
  5. 2025Depth Any Camera: Zero-Shot Metric Depth Estimation from Any Cameracited by 38
  6. 2023Self-supervised Monocular Depth Estimation: Let's Talk About The Weathercited by 33
  7. 2024Physical 3D Adversarial Attacks against Monocular Depth Estimation in Autonomous Drivingcited by 32
  8. 2024Hierarchical Temporal Context Learning for Camera-based Semantic Scene Completioncited by 32
  9. 2022Learning Optical Flow, Depth, and Scene Flow without Real-World Labelscited by 31
  10. 2025WonderTurbo: Generating Interactive 3D World in 0.72 Secondscited by 31
  11. 2024Cross-View Completion Models are Zero-shot Correspondence Estimatorscited by 30
  12. 2021Attention meets Geometry: Geometry Guided Spatial-Temporal Attention for Consistent Self-Supervised Monocular Depth Estimationcited by 27
  13. 2023On the Importance of Accurate Geometry Data for Dense 3D Vision Taskscited by 26
  14. 2025MVSAnywhere: Zero-Shot Multi-View Stereocited by 26
  15. 2023Learning to Fuse Monocular and Multi-view Cues for Multi-frame Depth Estimation in Dynamic Scenescited by 25
  16. 2022Are We Ready for Vision-Centric Driving Streaming Perception? The ASAP Benchmarkcited by 24
  17. 2023MAMo: Leveraging Memory and Attention for Monocular Video Depth Estimationcited by 24
  18. 2023RM-Depth: Unsupervised Learning of Recurrent Monocular Depth in Dynamic Scenescited by 22
  19. 2023SQLdepth: Generalizable Self-Supervised Fine-Structured Monocular Depth Estimationcited by 21
  20. 2022Self-Supervised Monocular Depth Estimation: Solving the Edge-Fattening Problemcited by 20
  21. 2023Dynamo-Depth: Fixing Unsupervised Depth Estimation for Dynamical Scenescited by 20
  22. 2025Jasmine: Harnessing Diffusion Prior for Self-supervised Depth Estimationcited by 20
  23. 2025Synthetic-to-Real Self-supervised Robust Depth Estimation via Learning with Motion and Structure Priorscited by 20
  24. 2022Is my Depth Ground-Truth Good Enough? HAMMER -- Highly Accurate Multi-Modal Dataset for DEnse 3D Scene Regressioncited by 18
  25. 2023MACARONS: Mapping And Coverage Anticipation with RGB Online Self-Supervisioncited by 18
  26. 2022Depth Field Networks for Generalizable Multi-view Scene Representationcited by 17
  27. 2023Adversarial Training of Self-supervised Monocular Depth Estimation against Physical-World Attackscited by 17
  28. 2023LightDepth: Single-View Depth Self-Supervision from Illumination Declinecited by 17
  29. 2022SelfD: Self-Learning Large-Scale Driving Policies From the Webcited by 16
  30. 2022Multi-Camera Collaborative Depth Prediction via Consistent Structure Estimationcited by 16
  31. 2022Consistent Direct Time-of-Flight Video Depth Super-Resolutioncited by 16
  32. 2023EGA-Depth: Efficient Guided Attention for Self-Supervised Multi-Camera Depth Estimationcited by 16
  33. 2023DualRefine: Self-Supervised Depth and Pose Estimation Through Iterative Epipolar Sampling and Refinement Toward Equilibriumcited by 16
  34. 2023(LC)2: LiDAR-Camera Loop Constraints For Cross-Modal Place Recognitioncited by 16
  35. 2023CVRecon: Rethinking 3D Geometric Feature Learning For Neural Reconstructioncited by 16
  36. 2023WeatherDepth: Curriculum Contrastive Learning for Self-Supervised Depth Estimation under Adverse Weather Conditionscited by 15
  37. 2024Deep Learning-based Depth Estimation Methods from Monocular Image and Videos: A Comprehensive Surveycited by 15
  38. 2023Self-Supervised Monocular Depth Estimation by Direction-aware Cumulative Convolution Networkcited by 14

50 most cited of 190 All 190 (JSON)