# Unsupervised Learning of 3D Scene Flow from Monocular Camera

arXiv: https://arxiv.org/abs/2206.03673 · HTML: https://mlsanity.com/paper/2206.03673/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- Builds on 27 of its 50 references that are in the map.
- First posted on arXiv 8 Jun 2022.

## Builds on (most cited first, 27 of 27)

- [PointNet: Deep Learning on Point Sets for 3D Classification and Segmentation](https://mlsanity.com/paper/1612.00593/) (2016)
- [PointNet++: Deep Hierarchical Feature Learning on Point Sets in a Metric Space](https://mlsanity.com/paper/1706.02413/) (2017)
- [FlowNet: Learning Optical Flow with Convolutional Networks](https://mlsanity.com/paper/1504.06852/) (2015)
- [A Large Dataset to Train Convolutional Networks for Disparity, Optical Flow, and Scene Flow Estimation](https://mlsanity.com/paper/1512.02134/) (2015)
- [FlowNet 2.0: Evolution of Optical Flow Estimation with Deep Networks](https://mlsanity.com/paper/1612.01925/) (2016)
- [GeoNet: Unsupervised Learning of Dense Depth, Optical Flow and Camera Pose](https://mlsanity.com/paper/1803.02276/) (2018)
- [Unsupervised Scale-consistent Depth and Ego-motion Learning from Monocular Video](https://mlsanity.com/paper/1908.10553/) (2019)
- [FlowNet3D: Learning Scene Flow in 3D Point Clouds](https://mlsanity.com/paper/1806.01411/) (2018)
- [Competitive Collaboration: Joint Unsupervised Learning of Depth, Camera Motion, Optical Flow and Motion Segmentation](https://mlsanity.com/paper/1805.09806/) (2018)
- [UnFlow: Unsupervised Learning of Optical Flow with a Bidirectional Census Loss](https://mlsanity.com/paper/1711.07837/) (2017)
- [Deep Parametric Continuous Convolutional Neural Networks](https://mlsanity.com/paper/2101.06742/) (2021)
- [SelFlow: Self-Supervised Learning of Optical Flow](https://mlsanity.com/paper/1904.09117/) (2019)
- [HPLFlowNet: Hierarchical Permutohedral Lattice FlowNet for Scene Flow Estimation on Large-scale Point Clouds](https://mlsanity.com/paper/1906.05332/) (2019)
- [Occlusion Aware Unsupervised Learning of Optical Flow](https://mlsanity.com/paper/1711.05890/) (2017)
- [Occlusions, Motion and Depth Boundaries with a Generic Network for Disparity, Optical Flow or Scene Flow Estimation](https://mlsanity.com/paper/1808.01838/) (2018)
- [Self-Supervised Monocular Scene Flow Estimation](https://mlsanity.com/paper/2004.04143/) (2020)
- [DDFlow: Learning Optical Flow with Unlabeled Data Distillation](https://mlsanity.com/paper/1902.09145/) (2019)
- [Deep Rigid Instance Scene Flow](https://mlsanity.com/paper/1904.08913/) (2019)
- [Learning Rigidity in Dynamic Scenes with a Moving Camera for 3D Motion Field Estimation](https://mlsanity.com/paper/1804.04259/) (2018)
- [SENSE: a Shared Encoder Network for Scene-flow Estimation](https://mlsanity.com/paper/1910.12361/) (2019)
- [Hierarchical Attention Learning of Scene Flow in 3D Point Clouds](https://mlsanity.com/paper/2010.05762/) (2020)
- [PWCLO-Net: Deep LiDAR Odometry in 3D Point Clouds Using Hierarchical Embedding Mask Optimization](https://mlsanity.com/paper/2012.00972/) (2020)
- [Unsupervised Learning of Depth, Optical Flow and Pose with Occlusion from 3D Geometry](https://mlsanity.com/paper/2003.00766/) (2020)
- [LiDAR-Flow: Dense Scene Flow Estimation from Sparse LiDAR and Stereo Images](https://mlsanity.com/paper/1910.14453/) (2019)
- [Spherical Interpolated Convolutional Network with Distance-Feature Density for 3D Semantic Segmentation of Point Clouds](https://mlsanity.com/paper/2011.13784/) (2020)
- [Unsupervised Learning of Monocular Depth and Ego-Motion Using Multiple Masks](https://mlsanity.com/paper/2104.00431/) (2021)
- [A Continuous Optimization Approach for Efficient and Accurate Scene Flow](https://mlsanity.com/paper/1607.07983/) (2016)

## Built on by (most cited first, 7 of 7)

- [RegFormer: An Efficient Projection-Aware Transformer Network for Large-Scale Point Cloud Registration](https://mlsanity.com/paper/2303.12384/) (2023)
- [What Matters for 3D Scene Flow Network](https://mlsanity.com/paper/2207.09143/) (2022)
- [DELFlow: Dense Efficient Learning of Scene Flow for Large-Scale Point Clouds](https://mlsanity.com/paper/2308.04383/) (2023)
- [RLSAC: Reinforcement Learning enhanced Sample Consensus for End-to-End Robust Estimation](https://mlsanity.com/paper/2308.05318/) (2023)
- [Pseudo-LiDAR for Visual Odometry](https://mlsanity.com/paper/2209.01567/) (2022)
- [LTS-NET: End-to-end Unsupervised Learning of Long-Term 3D Stable objects](https://mlsanity.com/paper/2301.03426/) (2023)
- [Long-Term 3D Point Tracking By Cost Volume Fusion](https://mlsanity.com/paper/2407.13337/) (2024)

## Abstract (arXiv)

Scene flow represents the motion of points in the 3D space, which is the counterpart of the optical flow that represents the motion of pixels in the 2D image. However, it is difficult to obtain the ground truth of scene flow in the real scenes, and recent studies are based on synthetic data for training. Therefore, how to train a scene flow network with unsupervised methods based on real-world data shows crucial significance. A novel unsupervised learning method for scene flow is proposed in this paper, which utilizes the images of two consecutive frames taken by monocular camera without the ground truth of scene flow for training. Our method realizes the goal that training scene flow network with real-world data, which bridges the gap between training data and test data and broadens the scope of available data for training. Unsupervised learning of scene flow in this paper mainly consists of two parts: (i) depth estimation and camera pose estimation, and (ii) scene flow estimation based on four different loss functions. Depth estimation and camera pose estimation obtain the depth maps and camera pose between two consecutive frames, which provide further information for the next scene flow estimation. After that, we used depth consistency loss, dynamic-static consistency loss, Chamfer loss, and Laplacian regularization loss to carry out unsupervised training of the scene flow network. To our knowledge, this is the first paper that realizes the unsupervised learning of 3D scene flow from monocular camera. The experiment results on KITTI show that our method for unsupervised learning of scene flow meets great performance compared to traditional methods Iterative Closest Point (ICP) and Fast Global Registration (FGR). The source code is available at: https://github.com/IRMVLab/3DUnMonoFlow.