# FutureDepth: Learning to Predict the Future Improves Video Depth Estimation

arXiv: https://arxiv.org/abs/2403.12953 · HTML: https://mlsanity.com/paper/2403.12953/

## Key facts

- Cited by 13 papers in the map (data through 14 Oct 2026).
- Builds on 44 of its 59 references that are in the map.
- First posted on arXiv 19 Mar 2024.

## Builds on (most cited first, 30 of 44)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Swin Transformer: Hierarchical Vision Transformer using Shifted Windows](https://mlsanity.com/paper/2103.14030/) (2021)
- [EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks](https://mlsanity.com/paper/1905.11946/) (2019)
- [SegFormer: Simple and Efficient Design for Semantic Segmentation with Transformers](https://mlsanity.com/paper/2105.15203/) (2021)
- [Vision Transformers for Dense Prediction](https://mlsanity.com/paper/2103.13413/) (2021)
- [Towards Robust Monocular Depth Estimation: Mixing Datasets for Zero-shot Cross-dataset Transfer](https://mlsanity.com/paper/1907.01341/) (2019)
- [Linformer: Self-Attention with Linear Complexity](https://mlsanity.com/paper/2006.04768/) (2020)
- [VideoMAE: Masked Autoencoders are Data-Efficient Learners for Self-Supervised Video Pre-Training](https://mlsanity.com/paper/2203.12602/) (2022)
- [Depth Map Prediction from a Single Image using a Multi-Scale Deep Network](https://mlsanity.com/paper/1406.2283/) (2014)
- [Learning Latent Dynamics for Planning from Pixels](https://mlsanity.com/paper/1811.04551/) (2018)
- [Mastering Diverse Domains through World Models](https://mlsanity.com/paper/2301.04104/) (2023)
- [A Large Dataset to Train Convolutional Networks for Disparity, Optical Flow, and Scene Flow Estimation](https://mlsanity.com/paper/1512.02134/) (2015)
- [Unsupervised Monocular Depth Estimation with Left-Right Consistency](https://mlsanity.com/paper/1609.03677/) (2016)
- [Deep Ordinal Regression Network for Monocular Depth Estimation](https://mlsanity.com/paper/1806.02446/) (2018)
- [AdaBins: Depth Estimation using Adaptive Bins](https://mlsanity.com/paper/2011.14141/) (2020)
- [VideoMAE V2: Scaling Video Masked Autoencoders with Dual Masking](https://mlsanity.com/paper/2303.16727/) (2023)
- [3D Packing for Self-Supervised Monocular Depth Estimation](https://mlsanity.com/paper/1905.02693/) (2019)
- [FlowFormer: A Transformer Architecture for Optical Flow](https://mlsanity.com/paper/2203.16194/) (2022)
- [Consistent Video Depth Estimation](https://mlsanity.com/paper/2004.15021/) (2020)
- [The Temporal Opportunist: Self-Supervised Multi-Frame Monocular Depth](https://mlsanity.com/paper/2104.14540/) (2021)
- [Robust Consistent Video Depth Estimation](https://mlsanity.com/paper/2012.05901/) (2020)
- [BinsFormer: Revisiting Adaptive Bins for Monocular Depth Estimation](https://mlsanity.com/paper/2204.00987/) (2022)
- [DepthFormer: Exploiting Long-Range Correlation and Local Information for Accurate Monocular Depth Estimation](https://mlsanity.com/paper/2203.14211/) (2022)
- [Attention Attention Everywhere: Monocular Depth Prediction with Skip Attention](https://mlsanity.com/paper/2210.09071/) (2022)
- [SimpleRecon: 3D Reconstruction Without 3D Convolutions](https://mlsanity.com/paper/2208.14743/) (2022)
- [iDisc: Internal Discretization for Monocular Depth Estimation](https://mlsanity.com/paper/2304.06334/) (2023)
- [Don't Forget The Past: Recurrent Depth Estimation from Monocular Video](https://mlsanity.com/paper/2001.02613/) (2020)
- [Neural RGB->D Sensing: Depth and Uncertainty from a Video Camera](https://mlsanity.com/paper/1901.02571/) (2019)
- [Recurrent Environment Simulators](https://mlsanity.com/paper/1704.02254/) (2017)

## Built on by (most cited first, 13 of 13)

- [DepthCrafter: Generating Consistent Long Depth Sequences for Open-world Videos](https://mlsanity.com/paper/2409.02095/) (2024)
- [Distilling Multi-modal Large Language Models for Autonomous Driving](https://mlsanity.com/paper/2501.09757/) (2025)
- [Video Depth without Video Models](https://mlsanity.com/paper/2411.19189/) (2024)
- [StereoDiff: Stereo-Diffusion Synergy for Video Depth Estimation](https://mlsanity.com/paper/2506.20756/) (2025)
- [Video Understanding: From Geometry and Semantics to Unified Models](https://mlsanity.com/paper/2603.17840/) (2026)
- [H3O: Hyper-Efficient 3D Occupancy Prediction with Heterogeneous Supervision](https://mlsanity.com/paper/2503.04059/) (2025)
- [SciFlow: Empowering Lightweight Optical Flow Models with Self-Cleaning Iterations](https://mlsanity.com/paper/2404.08135/) (2024)
- [DepthSync: Diffusion Guidance-Based Depth Synchronization for Scale- and Geometry-Consistent Video Depth Estimation](https://mlsanity.com/paper/2507.01603/) (2025)
- [Panoptic-Depth Forecasting](https://mlsanity.com/paper/2409.12008/) (2024)
- [BePo: Dual Representation for 3D Occupancy Prediction](https://mlsanity.com/paper/2506.07002/) (2025)
- [DySS: Dynamic Queries and State-Space Learning for Efficient 3D Object Detection from Multi-Camera Videos](https://mlsanity.com/paper/2506.10242/) (2025)
- [Hierarchical Awareness Adapters with Hybrid Pyramid Feature Fusion for Dense Depth Prediction](https://mlsanity.com/paper/2604.03339/) (2026)
- [GemDepth: Geometry-Embedded Features for 3D-Consistent Video Depth](https://mlsanity.com/paper/2605.10525/) (2026)

## Abstract (arXiv)

In this paper, we propose a novel video depth estimation approach, FutureDepth, which enables the model to implicitly leverage multi-frame and motion cues to improve depth estimation by making it learn to predict the future at training. More specifically, we propose a future prediction network, F-Net, which takes the features of multiple consecutive frames and is trained to predict multi-frame features one time step ahead iteratively. In this way, F-Net learns the underlying motion and correspondence information, and we incorporate its features into the depth decoding process. Additionally, to enrich the learning of multiframe correspondence cues, we further leverage a reconstruction network, R-Net, which is trained via adaptively masked auto-encoding of multiframe feature volumes. At inference time, both F-Net and R-Net are used to produce queries to work with the depth decoder, as well as a final refinement network. Through extensive experiments on several benchmarks, i.e., NYUDv2, KITTI, DDAD, and Sintel, which cover indoor, driving, and open-domain scenarios, we show that FutureDepth significantly improves upon baseline models, outperforms existing video depth estimation methods, and sets new state-of-the-art (SOTA) accuracy. Furthermore, FutureDepth is more efficient than existing SOTA video depth estimation models and has similar latencies when comparing to monocular models