mlsanityData to 7 Oct 2026

Learning Optical Flow, Depth, and Scene Flow without Real-World Labels

arXiv 2203.15089 ↗First posted 28 Mar 2022cs.CV cs.LGcited by 31

Not on the map. Why?

Abstract (arXiv)

Self-supervised monocular depth estimation enables robots to learn 3D perception from raw video streams. This scalable approach leverages projective geometry and ego-motion to learn via view synthesis, assuming the world is mostly static. Dynamic scenes, which are common in autonomous driving and human-robot interaction, violate this assumption. Therefore, they require modeling dynamic objects explicitly, for instance via estimating pixel-wise 3D motion, i.e. scene flow. However, the simultaneous self-supervised learning of depth and scene flow is ill-posed, as there are infinitely many combinations that result in the same 3D point. In this paper we propose DRAFT, a new method capable of jointly learning depth, optical flow, and scene flow by combining synthetic data with geometric self-supervision. Building upon the RAFT architecture, we learn optical flow as an intermediate task to bootstrap depth and scene flow learning via triangulation. Our algorithm also leverages temporal and geometric consistency losses across tasks to improve multi-task learning. Our DRAFT architecture simultaneously establishes a new state of the art in all three tasks in the self-supervised monocular setting on the standard KITTI benchmark. Project page: https://sites.google.com/tri.global/draft.

Key facts. Cited by 31 papers in the map (data through 7 Oct 2026). Builds on 52 of its 70 references that are in the map. First posted on arXiv 28 Mar 2022.

Builds on

52 of 70 references.

References · 52

  1. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  2. 2014Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translationcited by 4,016
  3. 2020RAFT: Recurrent All-Pairs Field Transforms for Optical Flowcited by 2,385
  4. 2015Spatial Transformer Networkscited by 1,423
  5. 2014Depth Map Prediction from a Single Image using a Multi-Scale Deep Networkcited by 1,264
  6. 2015FlowNet: Learning Optical Flow with Convolutional Networkscited by 1,054
  7. 2017PWC-Net: CNNs for Optical Flow Using Pyramid, Warping, and Cost Volumecited by 966
  8. 2015A Large Dataset to Train Convolutional Networks for Disparity, Optical Flow, and Scene Flow Estimationcited by 878
  9. 2016Unsupervised Monocular Depth Estimation with Left-Right Consistencycited by 819
  10. 2016FlowNet 2.0: Evolution of Optical Flow Estimation with Deep Networkscited by 780
  11. 2017Unsupervised Learning of Depth and Ego-Motion from Videocited by 778
  12. 20193D Packing for Self-Supervised Monocular Depth Estimationcited by 437
38 more
  1. 2016Optical Flow Estimation using a Spatial Pyramid Networkcited by 421
  2. 2020Virtual KITTI 2cited by 345
  3. 2018GeoNet: Unsupervised Learning of Dense Depth, Optical Flow and Camera Posecited by 341
  4. 2016Virtual Worlds as Proxy for Multi-Object Tracking Analysiscited by 340
  5. 2018FlowNet3D: Learning Scene Flow in 3D Point Cloudscited by 241
  6. 2021The Temporal Opportunist: Self-Supervised Multi-Frame Monocular Depthcited by 190
  7. 2019Depth from Videos in the Wild: Unsupervised Monocular Depth Learning from Unknown Camerascited by 170
  8. 2018DF-Net: Unsupervised Joint Learning of Depth and Flow using Cross-Task Consistencycited by 148
  9. 2020Semantically-Guided Representation Learning for Self-Supervised Monocular Depthcited by 134
  10. 2019Iterative Residual Refinement for Joint Optical Flow and Occlusion Estimationcited by 131
  11. 2019Self-supervised Learning with Geometric Constraints in Monocular Video: Connecting Flow, Depth, and Cameracited by 130
  12. 2019SelFlow: Self-Supervised Learning of Optical Flowcited by 128
  13. 2020What Matters in Unsupervised Optical Flowcited by 124
  14. 2020RAFT-3D: Scene Flow using Rigid-Motion Embeddingscited by 121
  15. 2016Back to Basics: Unsupervised Learning of Optical Flow via Brightness Constancy and Motion Smoothnesscited by 101
  16. 2018Every Pixel Counts ++: Joint Learning of Geometry and Motion with 3D Holistic Understandingcited by 101
  17. 2018Hierarchical Discrete Distribution Decomposition for Match Density Estimationcited by 93
  18. 2016Learning Dense Correspondence via 3D-guided Cycle Consistencycited by 91
  19. 2017SfM-Net: Learning of Structure and Motion from Videocited by 89
  20. 2018Deep Virtual Stereo Odometry: Leveraging Deep Depth Prediction for Monocular Direct Sparse Odometrycited by 85
  21. 2020Towards Better Generalization: Joint Depth-Pose Learning without PoseNetcited by 85
  22. 2020PV-RAFT: Point-Voxel Correlation Fields for Scene Flow Estimation of Point Cloudscited by 74
  23. 2018Occlusions, Motion and Depth Boundaries with a Generic Network for Disparity, Optical Flow or Scene Flow Estimationcited by 69
  24. 2020Self-Supervised Monocular Scene Flow Estimationcited by 64
  25. 2019Geometry-Aware Symmetric Domain Adaptation for Monocular Depth Estimationcited by 61
  26. 2021Full Surround Monodepth from Multiple Camerascited by 41
  27. 2018Learning Rigidity in Dynamic Scenes with a Moving Camera for 3D Motion Field Estimationcited by 38
  28. 2021Geometric Unsupervised Domain Adaptation for Semantic Segmentationcited by 33
  29. 2021Self-Supervised Multi-Frame Monocular Scene Flowcited by 29
  30. 2019Mono-SF: Multi-View Geometry Meets Single-View Depth for Monocular Scene Flow Estimation of Dynamic Traffic Scenescited by 27
  31. 2020SharinGAN: Combining Synthetic and Real Data for Unsupervised Geometry Estimationcited by 25
  32. 2018Every Pixel Counts: Unsupervised Geometry Learning with Holistic 3D Motion Understandingcited by 24
  33. 2021Monocular Depth Estimation through Virtual-world Supervision and Real-world SfM Self-Supervisioncited by 24
  34. 2020EffiScene: Efficient Per-Pixel Rigidity Inference for Unsupervised Joint Learning of Optical Flow, Depth, Camera Pose and Motion Segmentationcited by 22
  35. 2019PWOC-3D: Deep Occlusion-Aware End-to-End Scene Flow Estimationcited by 20
  36. 2020Neural Ray Surfaces for Self-Supervised Learning of Depth and Ego-motioncited by 20
  37. 2021Generalizing to the Open World: Deep Visual Odometry with Online Adaptationcited by 19
  38. 2020Deep Multi-view Depth Estimation with Predicted Uncertaintycited by 11

50 most cited of 52 All 52 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 3; 2023: 14; 2024: 8; 2025: 4; 2026: 2.

31 papers.

Sort:

By area:

  1. 2023Towards Zero-Shot Scale-Aware Monocular Depth Estimationcited by 100
  2. 2023Robust Monocular Depth Estimation under Challenging Conditionscited by 58
  3. 2023R3D3: Dense 3D Reconstruction of Dynamic Scenes from Multiple Camerascited by 39
  4. 2023FlowCam: Training Generalizable 3D Radiance Fields without Camera Poses via Pixel-Aligned Scene Flowcited by 38
  5. 2023Zero-Shot Metric Depth with a Field-of-View Conditioned Diffusion Modelcited by 29
  6. 2023RPEFlow: Multimodal Fusion of RGB-PointCloud-Event for Joint Optical Flow and Scene Flow Estimationcited by 26
  7. 2023Dynamo-Depth: Fixing Unsupervised Depth Estimation for Dynamical Scenescited by 20
  8. 2023EGA-Depth: Efficient Guided Attention for Self-Supervised Multi-Camera Depth Estimationcited by 16
  9. 2024Bring Event into RGB and LiDAR: Hierarchical Visual-Motion Fusion for Scene Flowcited by 15
  10. 2025Zero-Shot Novel View and Depth Synthesis with Multi-View Geometric Diffusioncited by 12
  11. 2026AnyView: Synthesizing Any Novel View in Dynamic Scenescited by 12
  12. 2022Mind The Edge: Refining Depth Edges in Sparsely-Supervised Monocular Depth Estimationcited by 8
19 more
  1. 2023DS-Depth: Dynamic and Static Depth Estimation via a Fusion Cost Volumecited by 8
  2. 2023Joint Self-supervised Depth and Optical Flow Estimation towards Dynamic Objectscited by 8
  3. 2024GRIN: Zero-Shot Metric Depth with Pixel-Level Diffusioncited by 6
  4. 2024SynFER: Towards Boosting Facial Expression Recognition with Synthetic Datacited by 6
  5. 2023DeLiRa: Self-Supervised Depth, Light, and Radiance Fieldscited by 5
  6. 2024TanDepth: Leveraging Global DEMs for Metric Monocular Depth Estimation in UAVscited by 4
  7. 2025ZeroVO: Visual Odometry with Minimal Assumptionscited by 4
  8. 2024Motor Focus: Fast Ego-Motion Prediction for Assistive Visual Navigationcited by 3
  9. 2024Multi-task Geometric Estimation of Depth and Surface Normal from Monocular 360deg Imagescited by 3
  10. 2025JointSplat: Probabilistic Joint Flow-Depth Optimization for Sparse-View Gaussian Splattingcited by 2
  11. 2023Do More With What You Have: Transferring Depth-Scale from Labeled to Unlabeled Domainscited by 1
  12. 2024GeoFormer: A Vision and Sequence Transformer-based Approach for Greenhouse Gas Monitoringcited by 1
  13. 2025Estimating Scene Flow in Robot Surroundings with Distributed Miniaturized Time-of-Flight Sensorscited by 1
  14. 2022A Compacted Structure for Cross-domain learning on Monocular Depth and Flow Estimationcited by 0
  15. 2022Neural Camera Modelscited by 0
  16. 2023Pose Constraints for Consistent Self-supervised Monocular Depth and Ego-motioncited by 0
  17. 2023Skin the sheep not only once: Reusing Various Depth Datasets to Drive the Learning of Optical Flowcited by 0
  18. 2024Neuromorphic spatiotemporal optical flow: Enabling ultrafast visual perception beyond human capabilitiescited by 0
  19. 2026PTC-Depth: Pose-Refined Monocular Depth Estimation with Temporal Consistencycited by 0