# Exploiting temporal consistency for real-time video depth estimation

arXiv: https://arxiv.org/abs/1908.03706 · HTML: https://mlsanity.com/paper/1908.03706/

## Key facts

- Cited by 61 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 10 Aug 2019.

## Built on by (most cited first, 30 of 61)

- [DepthCrafter: Generating Consistent Long Depth Sequences for Open-world Videos](https://mlsanity.com/paper/2409.02095/) (2024)
- [The Temporal Opportunist: Self-Supervised Multi-Frame Monocular Depth](https://mlsanity.com/paper/2104.14540/) (2021)
- [Learning Temporally Consistent Video Depth from Video Diffusion Priors](https://mlsanity.com/paper/2406.01493/) (2024)
- [Multi-Frame Self-Supervised Depth with Transformers](https://mlsanity.com/paper/2204.07616/) (2022)
- [Depth Any Video with Scalable Synthetic Data](https://mlsanity.com/paper/2410.10815/) (2024)
- [Disentangling Object Motion and Occlusion for Unsupervised Multi-frame Monocular Depth](https://mlsanity.com/paper/2203.15174/) (2022)
- [Hierarchical Temporal Context Learning for Camera-based Semantic Scene Completion](https://mlsanity.com/paper/2407.02077/) (2024)
- [Instance-wise Occlusion and Depth Orders in Natural Scenes](https://mlsanity.com/paper/2111.14562/) (2021)
- [Less is More: Consistent Video Depth Estimation with Masked Frames Modeling](https://mlsanity.com/paper/2208.00380/) (2022)
- [Video Depth without Video Models](https://mlsanity.com/paper/2411.19189/) (2024)
- [Attention meets Geometry: Geometry Guided Spatial-Temporal Attention for Consistent Self-Supervised Monocular Depth Estimation](https://mlsanity.com/paper/2110.08192/) (2021)
- [Temporally Consistent Online Depth Estimation in Dynamic Scenes](https://mlsanity.com/paper/2111.09337/) (2021)
- [SVDistNet: Self-Supervised Near-Field Distance Estimation on Surround View Fisheye Cameras](https://mlsanity.com/paper/2104.04420/) (2021)
- [MAMo: Leveraging Memory and Attention for Monocular Video Depth Estimation](https://mlsanity.com/paper/2307.14336/) (2023)
- [MACARONS: Mapping And Coverage Anticipation with RGB Online Self-Supervision](https://mlsanity.com/paper/2303.03315/) (2023)
- [Deep Learning-based Depth Estimation Methods from Monocular Image and Videos: A Comprehensive Survey](https://mlsanity.com/paper/2406.19675/) (2024)
- [FlashDepth: Real-time Streaming Video Depth Estimation at 2K Resolution](https://mlsanity.com/paper/2504.07093/) (2025)
- [Reconstructing 4D Spatial Intelligence: A Survey](https://mlsanity.com/paper/2507.21045/) (2025)
- [Advancing Self-supervised Monocular Depth Learning with Sparse LiDAR](https://mlsanity.com/paper/2109.09628/) (2021)
- [SRFNet: Monocular Depth Estimation with Fine-grained Structure via Spatial Reliability-oriented Fusion of Frames and Events](https://mlsanity.com/paper/2309.12842/) (2023)
- [FutureDepth: Learning to Predict the Future Improves Video Depth Estimation](https://mlsanity.com/paper/2403.12953/) (2024)
- [Crafting Monocular Cues and Velocity Guidance for Self-Supervised Multi-Frame Depth Learning](https://mlsanity.com/paper/2208.09170/) (2022)
- [Temporally Consistent Online Depth Estimation Using Point-Based Fusion](https://mlsanity.com/paper/2304.07435/) (2023)
- [Match Stereo Videos via Bidirectional Alignment](https://mlsanity.com/paper/2409.20283/) (2024)
- [M4Depth: Monocular depth estimation for autonomous vehicles in unseen environments](https://mlsanity.com/paper/2105.09847/) (2021)
- [Inflation with Diffusion: Efficient Temporal Adaptation for Text-to-Video Super-Resolution](https://mlsanity.com/paper/2401.10404/) (2024)
- [Mono-ViFI: A Unified Learning Framework for Self-supervised Single- and Multi-frame Monocular Depth Estimation](https://mlsanity.com/paper/2407.14126/) (2024)
- [DurLAR: A High-fidelity 128-channel LiDAR Dataset with Panoramic Ambient and Reflectivity Imagery for Multi-modal Autonomous Driving Applications](https://mlsanity.com/paper/2406.10068/) (2024)
- [NVDS+: Towards Efficient and Versatile Neural Stabilizer for Video Depth Estimation](https://mlsanity.com/paper/2307.08695/) (2023)
- [A Simple yet Effective Test-Time Adaptation for Zero-Shot Monocular Metric Depth Estimation](https://mlsanity.com/paper/2412.14103/) (2024)

## Abstract (arXiv)

Accuracy of depth estimation from static images has been significantly improved recently, by exploiting hierarchical features from deep convolutional neural networks (CNNs). Compared with static images, vast information exists among video frames and can be exploited to improve the depth estimation performance. In this work, we focus on exploring temporal information from monocular videos for depth estimation. Specifically, we take the advantage of convolutional long short-term memory (CLSTM) and propose a novel spatial-temporal CSLTM (ST-CLSTM) structure. Our ST-CLSTM structure can capture not only the spatial features but also the temporal correlations/consistency among consecutive video frames with negligible increase in computational cost. Additionally, in order to maintain the temporal consistency among the estimated depth frames, we apply the generative adversarial learning scheme and design a temporal consistency loss. The temporal consistency loss is combined with the spatial loss to update the model in an end-to-end fashion. By taking advantage of the temporal information, we build a video depth estimation framework that runs in real-time and generates visually pleasant results. Moreover, our approach is flexible and can be generalized to most existing depth estimation frameworks. Code is available at: https://tinyurl.com/STCLSTM