# RoMo: Robust Motion Segmentation Improves Structure from Motion

arXiv: https://arxiv.org/abs/2411.18650 · HTML: https://mlsanity.com/paper/2411.18650/

## Key facts

- Cited by 20 papers in the map (data through 14 Oct 2026).
- Builds on 24 of its 58 references that are in the map.
- First posted on arXiv 27 Nov 2024.

## Builds on (most cited first, 24 of 24)

- [High-Resolution Image Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2112.10752/) (2021)
- [Emerging Properties in Self-Supervised Vision Transformers](https://mlsanity.com/paper/2104.14294/) (2021)
- [3D Gaussian Splatting for Real-Time Radiance Field Rendering](https://mlsanity.com/paper/2308.04079/) (2023)
- [SAM 2: Segment Anything in Images and Videos](https://mlsanity.com/paper/2408.00714/) (2024)
- [RAFT: Recurrent All-Pairs Field Transforms for Optical Flow](https://mlsanity.com/paper/2003.12039/) (2020)
- [Vision Transformers for Dense Prediction](https://mlsanity.com/paper/2103.13413/) (2021)
- [DUSt3R: Geometric 3D Vision Made Easy](https://mlsanity.com/paper/2312.14132/) (2023)
- [DROID-SLAM: Deep Visual SLAM for Monocular, Stereo, and RGB-D Cameras](https://mlsanity.com/paper/2108.10869/) (2021)
- [MonST3R: A Simple Approach for Estimating Geometry in the Presence of Motion](https://mlsanity.com/paper/2410.03825/) (2024)
- [TAPIR: Tracking Any Point with per-frame Initialization and temporal Refinement](https://mlsanity.com/paper/2306.08637/) (2023)
- [Robust Dynamic Radiance Fields](https://mlsanity.com/paper/2301.02239/) (2023)
- [Self-supervised Video Object Segmentation by Motion Grouping](https://mlsanity.com/paper/2104.07658/) (2021)
- [The Surprising Effectiveness of Diffusion Models for Optical Flow and Monocular Depth Estimation](https://mlsanity.com/paper/2306.01923/) (2023)
- [RobustNeRF: Ignoring Distractors with Robust Losses](https://mlsanity.com/paper/2302.00833/) (2023)
- [ParticleSfM: Exploiting Dense Point Trajectories for Localizing Moving Cameras in the Wild](https://mlsanity.com/paper/2207.09137/) (2022)
- [Learning Smooth Neural Functions via Lipschitz Regularization](https://mlsanity.com/paper/2202.08345/) (2022)
- [NeRF On-the-go: Exploiting Uncertainty for Distractor-free NeRFs in the Wild](https://mlsanity.com/paper/2405.18715/) (2024)
- [Segmenting Moving Objects via an Object-Centric Layered Representation](https://mlsanity.com/paper/2207.02206/) (2022)
- [SpotlessSplats: Ignoring Distractors in 3D Gaussian Splatting](https://mlsanity.com/paper/2406.20055/) (2024)
- [LEAP-VO: Long-term Effective Any Point Tracking for Visual Odometry](https://mlsanity.com/paper/2401.01887/) (2024)
- [VideoCutLER: Surprisingly Simple Unsupervised Video Instance Segmentation](https://mlsanity.com/paper/2308.14710/) (2023)
- [EM-driven unsupervised learning for efficient motion segmentation](https://mlsanity.com/paper/2201.02074/) (2022)
- [Guided Slot Attention for Unsupervised Video Object Segmentation](https://mlsanity.com/paper/2303.08314/) (2023)
- [DymSLAM:4D Dynamic Scene Reconstruction Based on Geometrical Motion Segmentation](https://mlsanity.com/paper/2003.04569/) (2020)

## Built on by (most cited first, 20 of 20)

- [ViPE: Video Pose Engine for 3D Geometric Perception](https://mlsanity.com/paper/2508.10934/) (2025)
- [Easi3R: Estimating Disentangled Motion from DUSt3R Without Training](https://mlsanity.com/paper/2503.24391/) (2025)
- [VGGT4D: Mining Motion Cues in Visual Geometry Transformers for 4D Scene Reconstruction](https://mlsanity.com/paper/2511.19971/) (2025)
- [Dynamic Camera Poses and Where to Find Them](https://mlsanity.com/paper/2504.17788/) (2025)
- [iTACO: Interactable Digital Twins of Articulated Objects from Casually Captured RGBD Videos](https://mlsanity.com/paper/2506.08334/) (2025)
- [RobustSplat: Decoupling Densification and Dynamics for Transient-Free 3DGS](https://mlsanity.com/paper/2506.02751/) (2025)
- [Can Video Diffusion Model Reconstruct 4D Geometry?](https://mlsanity.com/paper/2503.21082/) (2025)
- [WildPose: A Unified Framework for Robust Pose Estimation in the Wild](https://mlsanity.com/paper/2605.12774/) (2026)
- [GMOS: Grounding Moving Object Segmentation in 3D Space and Time](https://mlsanity.com/paper/2605.30352/) (2026)
- [Gaussians on Fire: High-Frequency Reconstruction of Flames](https://mlsanity.com/paper/2511.22459/) (2025)
- [The Dynamic Prior: Understanding 3D Structures for Casual Dynamic Videos](https://mlsanity.com/paper/2512.05398/) (2025)
- [Sparse View Distractor-Free Gaussian Splatting](https://mlsanity.com/paper/2603.01603/) (2026)
- [FullCircle: Effortless 3D Reconstruction from Casual 360^circ Captures](https://mlsanity.com/paper/2603.22572/) (2026)
- [4DVGGT-D: 4D Visual Geometry Transformer with Improved Dynamic Depth Estimation](https://mlsanity.com/paper/2605.12027/) (2026)
- [RiGS: Rigid-aware 4D Gaussian Splatting from a Single Monocular Video](https://mlsanity.com/paper/2605.23672/) (2026)
- [Robust Multimodal Dynamic Object Segmentation](https://mlsanity.com/paper/2607.18153/) (2026)
- [ORBIT++: Benchmarking SfM in the Wild with 360deg Video](https://mlsanity.com/paper/2608.22039/) (2026)
- [DynEoMT: Learning Object Dynamicity from Online Segmentation Queries](https://mlsanity.com/paper/2609.14466/) (2026)
- [MAVP: Map-Aware Visuomotor Policies for Mobile Manipulation](https://mlsanity.com/paper/2609.26378/) (2026)
- [DispFlow-GS: Displacement Flow Supervision with Motion Disentangling for Monocular Deformable 3D Gaussian Splatting](https://mlsanity.com/paper/2609.36940/) (2026)

## Abstract (arXiv)

There has been extensive progress in the reconstruction and generation of 4D scenes from monocular casually-captured video. While these tasks rely heavily on known camera poses, the problem of finding such poses using structure-from-motion (SfM) often depends on robustly separating static from dynamic parts of a video. The lack of a robust solution to this problem limits the performance of SfM camera-calibration pipelines. We propose a novel approach to video-based motion segmentation to identify the components of a scene that are moving w.r.t. a fixed world frame. Our simple but effective iterative method, RoMo, combines optical flow and epipolar cues with a pre-trained video segmentation model. It outperforms unsupervised baselines for motion segmentation as well as supervised baselines trained from synthetic data. More importantly, the combination of an off-the-shelf SfM pipeline with our segmentation masks establishes a new state-of-the-art on camera calibration for scenes with dynamic content, outperforming existing methods by a substantial margin.