mlsanityData to 7 Oct 2026

DRO: Deep Recurrent Optimizer for Video to Depth

arXiv 2103.13201 ↗First posted 24 Mar 2021cs.CVcited by 10

Not on the map. Why?

Abstract (arXiv)

There are increasing interests of studying the video-to-depth (V2D) problem with machine learning techniques. While earlier methods directly learn a mapping from images to depth maps and camera poses, more recent works enforce multi-view geometry constraints through optimization embedded in the learning framework. This paper presents a novel optimization method based on recurrent neural networks to further exploit the potential of neural networks in V2D. Specifically, our neural optimizer alternately updates the depth and camera poses through iterations to minimize a feature-metric cost, and two gated recurrent units iteratively improve the results by tracing historical information. Extensive experimental results demonstrate that our method outperforms previous methods and is more efficient in computation and memory consumption than cost-volume-based methods. In particular, our self-supervised method outperforms previous supervised methods on the KITTI and ScanNet datasets. Our source code is available at https://github.com/aliyun/dro-sfm.

Key facts. Cited by 10 papers in the map (data through 7 Oct 2026). Builds on 37 of its 47 references that are in the map. First posted on arXiv 24 Mar 2021.

Builds on

37 of 47 references.

References · 37

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2017ScanNet: Richly-annotated 3D Reconstructions of Indoor Scenescited by 3,088
  3. 2020RAFT: Recurrent All-Pairs Field Transforms for Optical Flowcited by 2,385
  4. 2014Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modelingcited by 2,341
  5. 2014Depth Map Prediction from a Single Image using a Multi-Scale Deep Networkcited by 1,264
  6. 2018MVSNet: Depth Inference for Unstructured Multi-view Stereocited by 780
  7. 2017Unsupervised Learning of Depth and Ego-Motion from Videocited by 778
  8. 2018Deep Ordinal Regression Network for Monocular Depth Estimationcited by 721
  9. 2017OptNet: Differentiable Optimization as a Layer in Neural Networkscited by 588
  10. 2016Direct Sparse Odometrycited by 479
  11. 2019Differentiable Convex Optimization Layerscited by 458
  12. 20193D Packing for Self-Supervised Monocular Depth Estimationcited by 437
25 more
  1. 2019Cascade Cost Volume for High-Resolution Multi-View Stereo and Stereo Matchingcited by 414
  2. 2019From Big to Small: Multi-Scale Local Planar Guidance for Monocular Depth Estimationcited by 379
  3. 2018GeoNet: Unsupervised Learning of Dense Depth, Optical Flow and Camera Posecited by 341
  4. 2019Unsupervised Scale-consistent Depth and Ego-motion Learning from Monocular Videocited by 244
  5. 2018Competitive Collaboration: Joint Unsupervised Learning of Depth, Camera Motion, Optical Flow and Motion Segmentationcited by 222
  6. 2016DeMoN: Depth and Motion Network for Learning Monocular Stereocited by 200
  7. 2018DeepV2D: Video to Depth with Differentiable Structure from Motioncited by 185
  8. 2018Unsupervised Learning of Depth and Ego-Motion from Monocular Video Using 3D Geometric Constraintscited by 179
  9. 2018BA-Net: Dense Bundle Adjustment Networkcited by 173
  10. 2018Unsupervised Learning of Monocular Depth Estimation and Visual Odometry with Deep Feature Reconstructioncited by 159
  11. 2017Learning Depth from Monocular Videos using Direct Methodscited by 154
  12. 2018CodeSLAM - Learning a Compact, Optimisable Representation for Dense Visual SLAMcited by 151
  13. 2018DF-Net: Unsupervised Joint Learning of Depth and Flow using Cross-Task Consistencycited by 148
  14. 2020Self-Supervised Monocular Depth Estimation: Solving the Dynamic Object Problem by Semantic Guidancecited by 148
  15. 2017Solving ill-posed inverse problems using iterative deep neural networkscited by 145
  16. 2019Self-supervised Learning with Geometric Constraints in Monocular Video: Connecting Flow, Depth, and Cameracited by 130
  17. 2017Semi-Supervised Deep Learning for Monocular Depth Map Predictioncited by 109
  18. 2020Fast-MVSNet: Sparse-to-Dense Multi-View Stereo With Learned Propagation and Gauss-Newton Refinementcited by 105
  19. 2018DeepTAM: Deep Tracking and Mappingcited by 87
  20. 2018Deep Virtual Stereo Odometry: Leveraging Deep Depth Prediction for Monocular Direct Sparse Odometrycited by 85
  21. 2020Towards Better Generalization: Joint Depth-Pose Learning without PoseNetcited by 85
  22. 2019DeepSFM: Structure From Motion Via Deep Bundle Adjustmentcited by 70
  23. 2018End-to-End Learning of Motion Representation for Video Understandingcited by 43
  24. 2015Learning image representations tied to ego-motioncited by 41
  25. 2018LS-Net: Learning to Solve Nonlinear Least Squares for Monocular Stereocited by 28

Built on by

Cited by year

Citing papers per year: 2023: 3; 2024: 6; 2025: 1.

10 papers.

  1. 2024DepthSplat: Connecting Gaussian Splatting and Depthcited by 275
  2. 2023IEBins: Iterative Elastic Bins for Monocular Depth Estimationcited by 53
  3. 2023Deep Learning for Camera Calibration and Beyond: A Surveycited by 20
  4. 2024GGRt: Towards Pose-free Generalizable 3D Gaussian Splatting in Real-timecited by 20
  5. 2024DeepAAT: Deep Automated Aerial Triangulation for Fast UAV-based Mappingcited by 1
  6. 2024Revisit Self-supervised Depth Estimation with Local Structure-from-Motioncited by 1
  7. 2024HIVE: HIerarchical Volume Encoding for Neural Implicit Surface Reconstructioncited by 1
  8. 2024FBINeRF: Feature-Based Integrated Recurrent Network for Pinhole and Fisheye Neural Radiance Fieldscited by 1
  9. 2023Learning Monocular Depth in Dynamic Environment via Context-aware Temporal Attentioncited by 0
  10. 2025VIMD: Monocular Visual-Inertial Motion and Depth Estimationcited by 0