mlsanityData to 7 Oct 2026

Consistent Depth of Moving Objects in Video

arXiv 2108.01166 ↗First posted 2 Aug 2021cs.CV cs.GRcited by 56

Not on the map. Why?

Abstract (arXiv)

We present a method to estimate depth of a dynamic scene, containing arbitrary moving objects, from an ordinary video captured with a moving camera. We seek a geometrically and temporally consistent solution to this underconstrained problem: the depth predictions of corresponding points across frames should induce plausible, smooth motion in 3D. We formulate this objective in a new test-time training framework where a depth-prediction CNN is trained in tandem with an auxiliary scene-flow prediction MLP over the entire input video. By recursively unrolling the scene-flow prediction MLP over varying time steps, we compute both short-range scene flow to impose local smooth motion priors directly in 3D, and long-range scene flow to impose multi-view consistency constraints with wide baselines. We demonstrate accurate and temporally coherent results on a variety of challenging videos containing diverse moving objects (pets, people, cars), as well as camera motion. Our depth maps give rise to a number of depth-and-motion aware video editing effects such as object and lighting insertion.

Key facts. Cited by 56 papers in the map (data through 7 Oct 2026). Builds on 30 of its 53 references that are in the map. First posted on arXiv 2 Aug 2021.

Builds on

30 of 53 references.

References · 30

  1. 2017Mask R-CNNcited by 7,259
  2. 2020RAFT: Recurrent All-Pairs Field Transforms for Optical Flowcited by 2,385
  3. 2019Towards Robust Monocular Depth Estimation: Mixing Datasets for Zero-shot Cross-dataset Transfercited by 1,484
  4. 2014Depth Map Prediction from a Single Image using a Multi-Scale Deep Networkcited by 1,264
  5. 2015ORB-SLAM: a Versatile and Accurate Monocular SLAM Systemcited by 983
  6. 2016Unsupervised Monocular Depth Estimation with Left-Right Consistencycited by 819
  7. 2017Unsupervised Learning of Depth and Ego-Motion from Videocited by 778
  8. 2020Neural Scene Flow Fields for Space-Time View Synthesis of Dynamic Scenescited by 750
  9. 2018MegaDepth: Learning Single-View Depth Prediction from Internet Photoscited by 731
  10. 2018Deep Ordinal Regression Network for Monocular Depth Estimationcited by 721
  11. 2018GeoNet: Unsupervised Learning of Dense Depth, Optical Flow and Camera Posecited by 341
  12. 2020Consistent Video Depth Estimationcited by 220
18 more
  1. 2020Robust Consistent Video Depth Estimationcited by 188
  2. 2016Single-Image Depth Perception in the Wildcited by 179
  3. 2020Novel View Synthesis of Dynamic Scenes with Globally Coherent Depths from a Monocular Cameracited by 166
  4. 2018Depth Prediction Without the Sensors: Leveraging Structure for Unsupervised Learning from Monocular Videoscited by 155
  5. 2020Self-Supervised Monocular Depth Estimation: Solving the Dynamic Object Problem by Semantic Guidancecited by 148
  6. 2019Self-supervised Learning with Geometric Constraints in Monocular Video: Connecting Flow, Depth, and Cameracited by 130
  7. 2019Learning the Depths of Moving People by Watching Frozen Peoplecited by 120
  8. 2015The Fast Bilateral Solvercited by 86
  9. 2016VolumeDeform: Real-time Volumetric Non-rigid Reconstructioncited by 75
  10. 2019Web Stereo Video Supervision for Depth Prediction from Dynamic Scenescited by 73
  11. 2020Don't Forget The Past: Recurrent Depth Estimation from Monocular Videocited by 71
  12. 20204D Visualization of Dynamic Events from Unconstrained Multi-View Videoscited by 66
  13. 2020Layered Neural Rendering for Retiming People in Videocited by 51
  14. 2018Every Pixel Counts: Unsupervised Geometry Learning with Holistic 3D Motion Understandingcited by 24
  15. 2018Soccer on Your Tabletopcited by 20
  16. 2018A Benchmark and Evaluation of Non-Rigid Structure from Motioncited by 14
  17. 2017Fast Multi-frame Stereo Scene Flow with Motion Segmentationcited by 6
  18. 2016Dense Wide-Baseline Scene Flow From Two Handheld Video Camerascited by 3

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 1; 2022: 6; 2023: 11; 2024: 20; 2025: 14; 2026: 4.

56 papers.

Sort:

By area:

  1. 2024MonST3R: A Simple Approach for Estimating Geometry in the Presence of Motioncited by 431
  2. 2024Shape of Motion: 4D Reconstruction from a Single Videocited by 247
  3. 2024MegaSaM: Accurate, Fast, and Robust Structure and Motion from Casual Dynamic Videoscited by 224
  4. 2024DepthCrafter: Generating Consistent Long Depth Sequences for Open-world Videoscited by 202
  5. 2025Video Depth Anything: Consistent Depth Estimation for Super-Long Videoscited by 202
  6. 2024PhysDreamer: Physics-Based Interaction with 3D Objects via Video Generationcited by 185
  7. 2023SceneScape: Text-Driven Consistent Scene Generationcited by 160
  8. 2024Learning Temporally Consistent Video Depth from Video Diffusion Priorscited by 100
  9. 2024Stereo4D: Learning How Things Move in 3D from Internet Stereo Videoscited by 89
  10. 2024Align3R: Aligned Monocular Depth Estimation for Dynamic Videoscited by 79
  11. 2024TAPVid-3D: A Benchmark for Tracking Any Point in 3Dcited by 69
  12. 2024Depth Any Video with Scalable Synthetic Datacited by 57
38 more
  1. 2022Source-free Depth for Object Pop-outcited by 52
  2. 2024StereoCrafter: Diffusion-based Generation of Long and High-fidelity Stereoscopic 3D from Monocular Videoscited by 36
  3. 2025GeometryCrafter: Consistent Geometry Estimation for Open-world Videos with Diffusion Priorscited by 36
  4. 2023Uncertainty-aware State Space Transformer for Egocentric 3D Hand Trajectory Forecastingcited by 33
  5. 2022Less is More: Consistent Video Depth Estimation with Masked Frames Modelingcited by 30
  6. 2024Recent Trends in 3D Reconstruction of General Non-Rigid Scenescited by 28
  7. 20244D Gaussian Splatting in the Wild with Uncertainty-Aware Regularizationcited by 28
  8. 2024Video Depth without Video Modelscited by 28
  9. 2023Pseudo-Generalized Dynamic View Synthesis from a Videocited by 27
  10. 2022PlaneFormers: From Sparse View Planes to 3D Reconstructioncited by 26
  11. 2023DaRF: Boosting Radiance Fields from Sparse Inputs with Monocular Depth Adaptationcited by 25
  12. 2023Fast View Synthesis of Casual Videos with Soup-of-Planescited by 22
  13. 2024Deep Learning-based Depth Estimation Methods from Monocular Image and Videos: A Comprehensive Surveycited by 15
  14. 2024Fast Encoder-Based 3D from Casual Videos via Point Track Processingcited by 14
  15. 2025FlashDepth: Real-time Streaming Video Depth Estimation at 2K Resolutioncited by 14
  16. 2025DGS-LRM: Real-Time Deformable 3D Gaussian Reconstruction From Monocular Videoscited by 14
  17. 2021The Implicit Values of A Good Hand Shake: Handheld Multi-Frame Neural Depth Refinementcited by 13
  18. 2023Spatiotemporally Consistent HDR Indoor Lighting Estimationcited by 12
  19. 2022Towards 3D Scene Reconstruction from Locally Scale-Aligned Monocular Video Depthcited by 11
  20. 2023MoBluRF: Motion Deblurring Neural Radiance Fields for Blurry Monocular Videocited by 9
  21. 2022GeoRefine: Self-Supervised Online Depth Refinement for Accurate Dense Mappingcited by 8
  22. 2025Vid-CamEdit: Video Camera Trajectory Editing with Generative Rendering from Estimated Geometrycited by 7
  23. 2025C4D: 4D Made from 3D through Dual Correspondencescited by 7
  24. 2023NVDS+: Towards Efficient and Versatile Neural Stabilizer for Video Depth Estimationcited by 6
  25. 2024On Moving Object Segmentation from Monocular Video with Transformerscited by 6
  26. 2025E3D-Bench: A Benchmark for End-to-End 3D Geometric Foundation Modelscited by 6
  27. 2025StereoDiff: Stereo-Diffusion Synergy for Video Depth Estimationcited by 6
  28. 2026Video Understanding: From Geometry and Semantics to Unified Modelscited by 5
  29. 2025SVDC: Consistent Direct Time-of-Flight Video Depth Completion with Frequency Selective Fusioncited by 4
  30. 2024Mining Supervision for Dynamic Regions in Self-Supervised Monocular Depth Estimationcited by 3
  31. 2026Depth Completion as Parameter-Efficient Test-Time Adaptationcited by 3
  32. 2022Multi-view Monocular Depth and Uncertainty Prediction with Deep SfM in Dynamic Environmentscited by 2
  33. 2023Tracking Objects with 3D Representation from Videoscited by 2
  34. 2024RealSeal: Revolutionizing Media Authentication with Real-Time Realism Scoringcited by 2
  35. 2025GeoMan: Temporally Consistent Human Geometry Estimation using Image-to-Video Diffusioncited by 2
  36. 2025DepthSync: Diffusion Guidance-Based Depth Synchronization for Scale- and Geometry-Consistent Video Depth Estimationcited by 2
  37. 2023Edge-aware Consistent Stereo Video Depth Estimationcited by 1
  38. 2023LiDAR Meta Depth Completioncited by 1

50 most cited of 56 All 56 (JSON)