# Motion-based Object Segmentation based on Dense RGB-D Scene Flow

arXiv: https://arxiv.org/abs/1804.05195 · HTML: https://mlsanity.com/paper/1804.05195/

## Key facts

- Cited by 18 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 14 Apr 2018.

## Built on by (most cited first, 18 of 18)

- [Spring: A High-Resolution High-Detail Dataset and Benchmark for Scene Flow, Optical Flow and Stereo](https://mlsanity.com/paper/2303.01943/) (2023)
- [Neural Scene Flow Prior](https://mlsanity.com/paper/2111.01253/) (2021)
- [Fast Neural Scene Flow](https://mlsanity.com/paper/2304.09121/) (2023)
- [Dream2Flow: Bridging Video Generation and Open-World Manipulation with 3D Object Flow](https://mlsanity.com/paper/2512.24766/) (2025)
- [Category-Independent Articulated Object Tracking with Factor Graphs](https://mlsanity.com/paper/2205.03721/) (2022)
- [3D Object Detection with a Self-supervised Lidar Scene Flow Backbone](https://mlsanity.com/paper/2205.00705/) (2022)
- [Re-Evaluating LiDAR Scene Flow for Autonomous Driving](https://mlsanity.com/paper/2304.02150/) (2023)
- [RICE: Refining Instance Masks in Cluttered Environments with Graph Neural Networks](https://mlsanity.com/paper/2106.15711/) (2021)
- [Learning to Regrasp by Learning to Place](https://mlsanity.com/paper/2109.08817/) (2021)
- [OmniHang: Learning to Hang Arbitrary Objects using Contact Point Correspondences and Neural Collision Estimation](https://mlsanity.com/paper/2103.14283/) (2021)
- [Sequential Point Clouds: A Survey](https://mlsanity.com/paper/2204.09337/) (2022)
- [Fixing Malfunctional Objects With Learned Physical Simulation and Functional Prediction](https://mlsanity.com/paper/2205.02834/) (2022)
- [FedRSU: Federated Learning for Scene Flow Estimation on Roadside Units](https://mlsanity.com/paper/2401.12862/) (2024)
- [Fluid Viscosity Prediction Leveraging Computer Vision and Robot Interaction](https://mlsanity.com/paper/2308.02715/) (2023)
- [RMS-FlowNet++: Efficient and Robust Multi-Scale Scene Flow Estimation for Large-Scale Point Clouds](https://mlsanity.com/paper/2407.01129/) (2024)
- [Estimating Scene Flow in Robot Surroundings with Distributed Miniaturized Time-of-Flight Sensors](https://mlsanity.com/paper/2504.02439/) (2025)
- [EgoFlowNet: Non-Rigid Scene Flow from Point Clouds with Ego-Motion Support](https://mlsanity.com/paper/2407.02920/) (2024)
- [MotionBits: Video Segmentation through Motion-Level Analysis of Rigid Bodies](https://mlsanity.com/paper/2603.06846/) (2026)

## Abstract (arXiv)

Given two consecutive RGB-D images, we propose a model that estimates a dense 3D motion field, also known as scene flow. We take advantage of the fact that in robot manipulation scenarios, scenes often consist of a set of rigidly moving objects. Our model jointly estimates (i) the segmentation of the scene into an unknown but finite number of objects, (ii) the motion trajectories of these objects and (iii) the object scene flow. We employ an hourglass, deep neural network architecture. In the encoding stage, the RGB and depth images undergo spatial compression and correlation. In the decoding stage, the model outputs three images containing a per-pixel estimate of the corresponding object center as well as object translation and rotation. This forms the basis for inferring the object segmentation and final object scene flow. To evaluate our model, we generated a new and challenging, large-scale, synthetic dataset that is specifically targeted at robotic manipulation: It contains a large number of scenes with a very diverse set of simultaneously moving 3D objects and is recorded with a simulated, static RGB-D camera. In quantitative experiments, we show that we outperform state-of-the-art scene flow and motion-segmentation methods on this data set. In qualitative experiments, we show how our learned model transfers to challenging real-world scenes, visually generating better results than existing methods.