# Don't Forget The Past: Recurrent Depth Estimation from Monocular Video

arXiv: https://arxiv.org/abs/2001.02613 · HTML: https://mlsanity.com/paper/2001.02613/

## Key facts

- Cited by 71 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 8 Jan 2020.

## Built on by (most cited first, 30 of 71)

- [The Temporal Opportunist: Self-Supervised Multi-Frame Monocular Depth](https://mlsanity.com/paper/2104.14540/) (2021)
- [P3Depth: Monocular Depth Estimation with a Piecewise Planarity Prior](https://mlsanity.com/paper/2204.02091/) (2022)
- [Learning Temporally Consistent Video Depth from Video Diffusion Priors](https://mlsanity.com/paper/2406.01493/) (2024)
- [SimpleRecon: 3D Reconstruction Without 3D Convolutions](https://mlsanity.com/paper/2208.14743/) (2022)
- [Multi-Frame Self-Supervised Depth with Transformers](https://mlsanity.com/paper/2204.07616/) (2022)
- [Consistent Depth of Moving Objects in Video](https://mlsanity.com/paper/2108.01166/) (2021)
- [Fine-grained Semantics-aware Representation Enhancement for Self-supervised Monocular Depth Estimation](https://mlsanity.com/paper/2108.08829/) (2021)
- [Disentangling Object Motion and Occlusion for Unsupervised Multi-frame Monocular Depth](https://mlsanity.com/paper/2203.15174/) (2022)
- [Achieving Risk Control in Online Learning Settings](https://mlsanity.com/paper/2205.09095/) (2022)
- [Single Image Depth Estimation: An Overview](https://mlsanity.com/paper/2104.06456/) (2021)
- [Less is More: Consistent Video Depth Estimation with Masked Frames Modeling](https://mlsanity.com/paper/2208.00380/) (2022)
- [Cross-View Completion Models are Zero-shot Correspondence Estimators](https://mlsanity.com/paper/2412.09072/) (2024)
- [Attention meets Geometry: Geometry Guided Spatial-Temporal Attention for Consistent Self-Supervised Monocular Depth Estimation](https://mlsanity.com/paper/2110.08192/) (2021)
- [MVSAnywhere: Zero-Shot Multi-View Stereo](https://mlsanity.com/paper/2503.22430/) (2025)
- [MAMo: Leveraging Memory and Attention for Monocular Video Depth Estimation](https://mlsanity.com/paper/2307.14336/) (2023)
- [SQLdepth: Generalizable Self-Supervised Fine-Structured Monocular Depth Estimation](https://mlsanity.com/paper/2309.00526/) (2023)
- [Self-Supervised Monocular Depth Estimation: Solving the Edge-Fattening Problem](https://mlsanity.com/paper/2210.00411/) (2022)
- [MACARONS: Mapping And Coverage Anticipation with RGB Online Self-Supervision](https://mlsanity.com/paper/2303.03315/) (2023)
- [UncLe-SLAM: Uncertainty Learning for Dense Neural SLAM](https://mlsanity.com/paper/2306.11048/) (2023)
- [Monocular Visual-Inertial Depth Estimation](https://mlsanity.com/paper/2303.12134/) (2023)
- [Consistent Direct Time-of-Flight Video Depth Super-Resolution](https://mlsanity.com/paper/2211.08658/) (2022)
- [EGA-Depth: Efficient Guided Attention for Self-Supervised Multi-Camera Depth Estimation](https://mlsanity.com/paper/2304.03369/) (2023)
- [DualRefine: Self-Supervised Depth and Pose Estimation Through Iterative Epipolar Sampling and Refinement Toward Equilibrium](https://mlsanity.com/paper/2304.03560/) (2023)
- [Deep Learning-based Depth Estimation Methods from Monocular Image and Videos: A Comprehensive Survey](https://mlsanity.com/paper/2406.19675/) (2024)
- [FlashDepth: Real-time Streaming Video Depth Estimation at 2K Resolution](https://mlsanity.com/paper/2504.07093/) (2025)
- [Reconstructing 4D Spatial Intelligence: A Survey](https://mlsanity.com/paper/2507.21045/) (2025)
- [FutureDepth: Learning to Predict the Future Improves Video Depth Estimation](https://mlsanity.com/paper/2403.12953/) (2024)
- [X-Distill: Improving Self-Supervised Monocular Depth via Cross-Task Distillation](https://mlsanity.com/paper/2110.12516/) (2021)
- [A Confidence-based Iterative Solver of Depths and Surface Normals for Deep Multi-view Stereo](https://mlsanity.com/paper/2201.07609/) (2022)
- [Crafting Monocular Cues and Velocity Guidance for Self-Supervised Multi-Frame Depth Learning](https://mlsanity.com/paper/2208.09170/) (2022)

## Abstract (arXiv)

Autonomous cars need continuously updated depth information. Thus far, depth is mostly estimated independently for a single frame at a time, even if the method starts from video input. Our method produces a time series of depth maps, which makes it an ideal candidate for online learning approaches. In particular, we put three different types of depth estimation (supervised depth prediction, self-supervised depth prediction, and self-supervised depth completion) into a common framework. We integrate the corresponding networks with a ConvLSTM such that the spatiotemporal structures of depth across frames can be exploited to yield a more accurate depth estimation. Our method is flexible. It can be applied to monocular videos only or be combined with different types of sparse depth patterns. We carefully study the architecture of the recurrent network and its training strategy. We are first to successfully exploit recurrent networks for real-time self-supervised monocular depth estimation and completion. Extensive experiments show that our recurrent method outperforms its image-based counterpart consistently and significantly in both self-supervised scenarios. It also outperforms previous depth estimation methods of the three popular groups. Please refer to https://www.trace.ethz.ch/publications/2020/rec_depth_estimation/ for details.