# Temporally Consistent Online Depth Estimation in Dynamic Scenes

arXiv: https://arxiv.org/abs/2111.09337 · HTML: https://mlsanity.com/paper/2111.09337/

## Key facts

- Cited by 26 papers in the map (data through 14 Oct 2026).
- Builds on 25 of its 48 references that are in the map.
- First posted on arXiv 17 Nov 2021.

## Builds on (most cited first, 25 of 25)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [RAFT: Recurrent All-Pairs Field Transforms for Optical Flow](https://mlsanity.com/paper/2003.12039/) (2020)
- [Spatial Transformer Networks](https://mlsanity.com/paper/1506.02025/) (2015)
- [Deep High-Resolution Representation Learning for Visual Recognition](https://mlsanity.com/paper/1908.07919/) (2019)
- [A Large Dataset to Train Convolutional Networks for Disparity, Optical Flow, and Scene Flow Estimation](https://mlsanity.com/paper/1512.02134/) (2015)
- [Accelerating 3D Deep Learning with PyTorch3D](https://mlsanity.com/paper/2007.08501/) (2020)
- [TartanAir: A Dataset to Push the Limits of Visual SLAM](https://mlsanity.com/paper/2003.14338/) (2020)
- [Pyramid Stereo Matching Network](https://mlsanity.com/paper/1803.08669/) (2018)
- [Sparsity Invariant CNNs](https://mlsanity.com/paper/1708.06500/) (2017)
- [FlowNet3D: Learning Scene Flow in 3D Point Clouds](https://mlsanity.com/paper/1806.01411/) (2018)
- [Group-wise Correlation Stereo Network](https://mlsanity.com/paper/1903.04025/) (2019)
- [Consistent Video Depth Estimation](https://mlsanity.com/paper/2004.15021/) (2020)
- [Revisiting Stereo Depth Estimation From a Sequence-to-Sequence Perspective with Transformers](https://mlsanity.com/paper/2011.02910/) (2020)
- [Robust Consistent Video Depth Estimation](https://mlsanity.com/paper/2012.05901/) (2020)
- [RAFT-3D: Scene Flow using Rigid-Motion Embeddings](https://mlsanity.com/paper/2012.00726/) (2020)
- [HITNet: Hierarchical Iterative Tile Refinement Network for Real-time Stereo Matching](https://mlsanity.com/paper/2007.12140/) (2020)
- [VolumeDeform: Real-time Volumetric Non-rigid Reconstruction](https://mlsanity.com/paper/1603.08161/) (2016)
- [Neural RGB->D Sensing: Depth and Uncertainty from a Video Camera](https://mlsanity.com/paper/1901.02571/) (2019)
- [Exploiting temporal consistency for real-time video depth estimation](https://mlsanity.com/paper/1908.03706/) (2019)
- [Deep Rigid Instance Scene Flow](https://mlsanity.com/paper/1904.08913/) (2019)
- [E-DSSR: Efficient Dynamic Surgical Scene Reconstruction with Transformer-based Stereoscopic Depth Perception](https://mlsanity.com/paper/2107.00229/) (2021)
- [A Robotic 3D Perception System for Operating Room Environment Awareness](https://mlsanity.com/paper/2003.09487/) (2020)
- [Temporally Consistent Depth Prediction with Flow-Guided Memory Units](https://mlsanity.com/paper/1909.07074/) (2019)

## Built on by (most cited first, 26 of 26)

- [DepthCrafter: Generating Consistent Long Depth Sequences for Open-world Videos](https://mlsanity.com/paper/2409.02095/) (2024)
- [DynamicStereo: Consistent Dynamic Depth from Stereo Videos](https://mlsanity.com/paper/2305.02296/) (2023)
- [Stereo4D: Learning How Things Move in 3D from Internet Stereo Videos](https://mlsanity.com/paper/2412.09621/) (2024)
- [Align3R: Aligned Monocular Depth Estimation for Dynamic Videos](https://mlsanity.com/paper/2412.03079/) (2024)
- [Context-Enhanced Stereo Transformer](https://mlsanity.com/paper/2210.11719/) (2022)
- [Tracking and Mapping in Medical Computer Vision: A Review](https://mlsanity.com/paper/2310.11475/) (2023)
- [Video Depth without Video Models](https://mlsanity.com/paper/2411.19189/) (2024)
- [Match-Stereo-Videos: Bidirectional Alignment for Consistent Dynamic Stereo Matching](https://mlsanity.com/paper/2403.10755/) (2024)
- [CaRTS: Causality-driven Robot Tool Segmentation from Vision and Kinematics Data](https://mlsanity.com/paper/2203.09475/) (2022)
- [Consistent Direct Time-of-Flight Video Depth Super-Resolution](https://mlsanity.com/paper/2211.08658/) (2022)
- [FlashDepth: Real-time Streaming Video Depth Estimation at 2K Resolution](https://mlsanity.com/paper/2504.07093/) (2025)
- [Reconstructing 4D Spatial Intelligence: A Survey](https://mlsanity.com/paper/2507.21045/) (2025)
- [Temporally Consistent Online Depth Estimation Using Point-Based Fusion](https://mlsanity.com/paper/2304.07435/) (2023)
- [Match Stereo Videos via Bidirectional Alignment](https://mlsanity.com/paper/2409.20283/) (2024)
- [Stereo Any Video: Temporally Consistent Stereo Matching](https://mlsanity.com/paper/2503.05549/) (2025)
- [NVDS+: Towards Efficient and Versatile Neural Stabilizer for Video Depth Estimation](https://mlsanity.com/paper/2307.08695/) (2023)
- [StereoDiff: Stereo-Diffusion Synergy for Video Depth Estimation](https://mlsanity.com/paper/2506.20756/) (2025)
- [Video Understanding: From Geometry and Semantics to Unified Models](https://mlsanity.com/paper/2603.17840/) (2026)
- [TAToo: Vision-based Joint Tracking of Anatomy and Tool for Skull-base Surgery](https://mlsanity.com/paper/2212.14131/) (2022)
- [Consistent Time-of-Flight Depth Denoising via Graph-Informed Geometric Attention](https://mlsanity.com/paper/2506.23542/) (2025)
- [DO3D: Self-supervised Learning of Decomposed Object-aware 3D Motion and Depth from Monocular Videos](https://mlsanity.com/paper/2403.05895/) (2024)
- [Into the Fog: Evaluating Robustness of Multiple Object Tracking](https://mlsanity.com/paper/2404.10534/) (2024)
- [Back2Color: Domain-Adaptive Synthetic-to-Real Monocular Depth Estimation for Dynamic Traffic Scenes](https://mlsanity.com/paper/2406.07741/) (2024)
- [Online Video Depth Anything: Temporally-Consistent Depth Prediction with Low Memory Consumption](https://mlsanity.com/paper/2510.09182/) (2025)
- [PPMStereo: Pick-and-Play Memory Construction for Consistent Dynamic Stereo Matching](https://mlsanity.com/paper/2510.20178/) (2025)
- [TiS-TSL: Image-Label Supervised Surgical Video Stereo Matching via Time-Switchable Teacher-Student Learning](https://mlsanity.com/paper/2511.06817/) (2025)

## Abstract (arXiv)

Temporally consistent depth estimation is crucial for online applications such as augmented reality. While stereo depth estimation has received substantial attention as a promising way to generate 3D information, there is relatively little work focused on maintaining temporal stability. Indeed, based on our analysis, current techniques still suffer from poor temporal consistency. Stabilizing depth temporally in dynamic scenes is challenging due to concurrent object and camera motion. In an online setting, this process is further aggravated because only past frames are available. We present a framework named Consistent Online Dynamic Depth (CODD) to produce temporally consistent depth estimates in dynamic scenes in an online setting. CODD augments per-frame stereo networks with novel motion and fusion networks. The motion network accounts for dynamics by predicting a per-pixel SE3 transformation and aligning the observations. The fusion network improves temporal depth consistency by aggregating the current and past estimates. We conduct extensive experiments and demonstrate quantitatively and qualitatively that CODD outperforms competing methods in terms of temporal consistency and performs on par in terms of per-frame accuracy.