# Self-Supervised Multi-Frame Monocular Scene Flow

arXiv: https://arxiv.org/abs/2105.02216 · HTML: https://mlsanity.com/paper/2105.02216/

## Key facts

- Cited by 29 papers in the map (data through 14 Oct 2026).
- Builds on 44 of its 85 references that are in the map.
- First posted on arXiv 5 May 2021.

## Builds on (most cited first, 30 of 44)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [nuScenes: A multimodal dataset for autonomous driving](https://mlsanity.com/paper/1903.11027/) (2019)
- [Convolutional LSTM Network: A Machine Learning Approach for Precipitation Nowcasting](https://mlsanity.com/paper/1506.04214/) (2015)
- [Spatial Transformer Networks](https://mlsanity.com/paper/1506.02025/) (2015)
- [PWC-Net: CNNs for Optical Flow Using Pyramid, Warping, and Cost Volume](https://mlsanity.com/paper/1709.02371/) (2017)
- [A Large Dataset to Train Convolutional Networks for Disparity, Optical Flow, and Scene Flow Estimation](https://mlsanity.com/paper/1512.02134/) (2015)
- [Unsupervised Monocular Depth Estimation with Left-Right Consistency](https://mlsanity.com/paper/1609.03677/) (2016)
- [Open3D: A Modern Library for 3D Data Processing](https://mlsanity.com/paper/1801.09847/) (2018)
- [GeoNet: Unsupervised Learning of Dense Depth, Optical Flow and Camera Pose](https://mlsanity.com/paper/1803.02276/) (2018)
- [Softmax Splatting for Video Frame Interpolation](https://mlsanity.com/paper/2003.05534/) (2020)
- [FlowNet3D: Learning Scene Flow in 3D Point Clouds](https://mlsanity.com/paper/1806.01411/) (2018)
- [Competitive Collaboration: Joint Unsupervised Learning of Depth, Camera Motion, Optical Flow and Motion Segmentation](https://mlsanity.com/paper/1805.09806/) (2018)
- [UnFlow: Unsupervised Learning of Optical Flow with a Bidirectional Census Loss](https://mlsanity.com/paper/1711.07837/) (2017)
- [DF-Net: Unsupervised Joint Learning of Depth and Flow using Cross-Task Consistency](https://mlsanity.com/paper/1809.01649/) (2018)
- [Self-supervised Learning with Geometric Constraints in Monocular Video: Connecting Flow, Depth, and Camera](https://mlsanity.com/paper/1907.05820/) (2019)
- [SelFlow: Self-Supervised Learning of Optical Flow](https://mlsanity.com/paper/1904.09117/) (2019)
- [What Matters in Unsupervised Optical Flow](https://mlsanity.com/paper/2006.04902/) (2020)
- [HPLFlowNet: Hierarchical Permutohedral Lattice FlowNet for Scene Flow Estimation on Large-scale Point Clouds](https://mlsanity.com/paper/1906.05332/) (2019)
- [Every Pixel Counts ++: Joint Learning of Geometry and Motion with 3D Holistic Understanding](https://mlsanity.com/paper/1810.06125/) (2018)
- [Just Go with the Flow: Self-Supervised Scene Flow Estimation](https://mlsanity.com/paper/1912.00497/) (2019)
- [Web Stereo Video Supervision for Depth Prediction from Dynamic Scenes](https://mlsanity.com/paper/1904.11112/) (2019)
- [Occlusions, Motion and Depth Boundaries with a Generic Network for Disparity, Optical Flow or Scene Flow Estimation](https://mlsanity.com/paper/1808.01838/) (2018)
- [Self-Supervised Monocular Scene Flow Estimation](https://mlsanity.com/paper/2004.04143/) (2020)
- [DDFlow: Learning Optical Flow with Unlabeled Data Distillation](https://mlsanity.com/paper/1902.09145/) (2019)
- [FlowNet3D++: Geometric Losses For Deep Scene Flow Estimation](https://mlsanity.com/paper/1912.01438/) (2019)
- [Deep Rigid Instance Scene Flow](https://mlsanity.com/paper/1904.08913/) (2019)
- [Learning Rigidity in Dynamic Scenes with a Moving Camera for 3D Motion Field Estimation](https://mlsanity.com/paper/1804.04259/) (2018)
- [SENSE: a Shared Encoder Network for Scene-flow Estimation](https://mlsanity.com/paper/1910.12361/) (2019)
- [A Fusion Approach for Multi-Frame Optical Flow Estimation](https://mlsanity.com/paper/1810.10066/) (2018)
- [Mono-SF: Multi-View Geometry Meets Single-View Depth for Monocular Scene Flow Estimation of Dynamic Traffic Scenes](https://mlsanity.com/paper/1908.06316/) (2019)

## Built on by (most cited first, 29 of 29)

- [VideoFlow: Exploiting Temporal Cues for Multi-frame Optical Flow Estimation](https://mlsanity.com/paper/2303.08340/) (2023)
- [Multi-Frame Self-Supervised Depth with Transformers](https://mlsanity.com/paper/2204.07616/) (2022)
- [IFOR: Iterative Flow Minimization for Robotic Object Rearrangement](https://mlsanity.com/paper/2202.00732/) (2022)
- [Fast Neural Scene Flow](https://mlsanity.com/paper/2304.09121/) (2023)
- [Learning Optical Flow, Depth, and Scene Flow without Real-World Labels](https://mlsanity.com/paper/2203.15089/) (2022)
- [What Matters for 3D Scene Flow Network](https://mlsanity.com/paper/2207.09143/) (2022)
- [Zero-Shot Monocular Scene Flow Estimation in the Wild](https://mlsanity.com/paper/2501.10357/) (2025)
- [Dynamo-Depth: Fixing Unsupervised Depth Estimation for Dynamical Scenes](https://mlsanity.com/paper/2310.18887/) (2023)
- [Bring Event into RGB and LiDAR: Hierarchical Visual-Motion Fusion for Scene Flow](https://mlsanity.com/paper/2403.07432/) (2024)
- [M-FUSE: Multi-frame Fusion for Scene Flow Estimation](https://mlsanity.com/paper/2207.05704/) (2022)
- [Learning Optical Flow and Scene Flow with Bidirectional Camera-LiDAR Fusion](https://mlsanity.com/paper/2303.12017/) (2023)
- [DELTAv2: Accelerating Dense 3D Tracking](https://mlsanity.com/paper/2508.01170/) (2025)
- [ADFactory: An Effective Framework for Generalizing Optical Flow with Nerf](https://mlsanity.com/paper/2311.04246/) (2023)
- [DS-Depth: Dynamic and Static Depth Estimation via a Fusion Cost Volume](https://mlsanity.com/paper/2308.07225/) (2023)
- [RAFT-MSF: Self-Supervised Monocular Scene Flow using Recurrent Optimizer](https://mlsanity.com/paper/2205.01568/) (2022)
- [3D Scene Flow Estimation on Pseudo-LiDAR: Bridging the Gap on Estimating Point Motion](https://mlsanity.com/paper/2209.13130/) (2022)
- [Learning to Estimate Single-View Volumetric Flow Motions without 3D Supervision](https://mlsanity.com/paper/2302.14470/) (2023)
- [EMR-MSF: Self-Supervised Recurrent Monocular Scene Flow Exploiting Ego-Motion Rigidity](https://mlsanity.com/paper/2309.01296/) (2023)
- [Time-to-Label: Temporal Consistency for Self-Supervised Monocular 3D Object Detection](https://mlsanity.com/paper/2203.02193/) (2022)
- [Learning Scene Flow in 3D Point Clouds with Noisy Pseudo Labels](https://mlsanity.com/paper/2203.12655/) (2022)
- [ContrastMotion: Self-supervised Scene Motion Learning for Large-Scale LiDAR Point Clouds](https://mlsanity.com/paper/2304.12589/) (2023)
- [Self-Supervised Multi-Frame Neural Scene Flow](https://mlsanity.com/paper/2403.16116/) (2024)
- [Self-Assessed Generation: Trustworthy Label Generation for Optical Flow and Stereo Matching in Real-world](https://mlsanity.com/paper/2410.10453/) (2024)
- [MonoPLFlowNet: Permutohedral Lattice FlowNet for Real-Scale 3D Scene FlowEstimation with Monocular Images](https://mlsanity.com/paper/2111.12325/) (2021)
- [DEFLOW: Self-supervised 3D Motion Estimation of Debris Flow](https://mlsanity.com/paper/2304.02569/) (2023)
- [Human Insights Driven Latent Space for Different Driving Perspectives: A Unified Encoder for Efficient Multi-Task Inference](https://mlsanity.com/paper/2409.10095/) (2024)
- [Self-SuperFlow: Self-supervised Scene Flow Prediction in Stereo Sequences](https://mlsanity.com/paper/2206.15296/) (2022)
- [Pseudo-Stereo Inputs: A Solution to the Occlusion Challenge in Self-Supervised Stereo Matching](https://mlsanity.com/paper/2410.02534/) (2024)
- [RAFT-MSF++: Temporal Geometry-Motion Feature Fusion for Self-Supervised Monocular Scene Flow](https://mlsanity.com/paper/2604.19349/) (2026)

## Abstract (arXiv)

Estimating 3D scene flow from a sequence of monocular images has been gaining increased attention due to the simple, economical capture setup. Owing to the severe ill-posedness of the problem, the accuracy of current methods has been limited, especially that of efficient, real-time approaches. In this paper, we introduce a multi-frame monocular scene flow network based on self-supervised learning, improving the accuracy over previous networks while retaining real-time efficiency. Based on an advanced two-frame baseline with a split-decoder design, we propose (i) a multi-frame model using a triple frame input and convolutional LSTM connections, (ii) an occlusion-aware census loss for better accuracy, and (iii) a gradient detaching strategy to improve training stability. On the KITTI dataset, we observe state-of-the-art accuracy among monocular scene flow methods based on self-supervised learning.