# Motion Segmentation from a Moving Monocular Camera

arXiv: https://arxiv.org/abs/2309.13772 · HTML: https://mlsanity.com/paper/2309.13772/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- Builds on 19 of its 47 references that are in the map.
- First posted on arXiv 24 Sep 2023.

## Builds on (most cited first, 19 of 19)

- [Grounding DINO: Marrying DINO with Grounded Pre-Training for Open-Set Object Detection](https://mlsanity.com/paper/2303.05499/) (2023)
- [RAFT: Recurrent All-Pairs Field Transforms for Optical Flow](https://mlsanity.com/paper/2003.12039/) (2020)
- [Recognize Anything: A Strong Image Tagging Model](https://mlsanity.com/paper/2306.03514/) (2023)
- [TAPIR: Tracking Any Point with per-frame Initialization and temporal Refinement](https://mlsanity.com/paper/2306.08637/) (2023)
- [Particle Video Revisited: Tracking Through Occlusions Using Point Trajectories](https://mlsanity.com/paper/2204.04153/) (2022)
- [Segment and Track Anything](https://mlsanity.com/paper/2305.06558/) (2023)
- [Decoupling Features in Hierarchical Propagation for Video Object Segmentation](https://mlsanity.com/paper/2210.09782/) (2022)
- [Segment Anything Meets Point Tracking](https://mlsanity.com/paper/2307.01197/) (2023)
- [It's Moving! A Probabilistic Model for Causal Motion Segmentation in Moving Camera Videos](https://mlsanity.com/paper/1604.00136/) (2016)
- [Towards Segmenting Anything That Moves](https://mlsanity.com/paper/1902.03715/) (2019)
- [EM-driven unsupervised learning for efficient motion segmentation](https://mlsanity.com/paper/2201.02074/) (2022)
- [Disentangling Architecture and Training for Optical Flow](https://mlsanity.com/paper/2203.10712/) (2022)
- [EpO-Net: Exploiting Geometric Constraints on Dense Trajectories for Motion Saliency](https://mlsanity.com/paper/1909.13258/) (2019)
- [Monocular Instance Motion Segmentation for Autonomous Driving: KITTI InstanceMotSeg Dataset and Multi-task Baseline](https://mlsanity.com/paper/2008.07008/) (2020)
- [Learning Independent Object Motion from Unlabelled Stereoscopic Videos](https://mlsanity.com/paper/1901.01971/) (2019)
- [From SLAM to Situational Awareness: Challenges and Survey](https://mlsanity.com/paper/2110.00273/) (2021)
- [RST-MODNet: Real-time Spatio-temporal Moving Object Detection for Autonomous Driving](https://mlsanity.com/paper/1912.00438/) (2019)
- [Motion Segmentation by Exploiting Complementary Geometric Models](https://mlsanity.com/paper/1804.02142/) (2018)
- [Deep Learning for Robust Motion Segmentation with Non-Static Cameras](https://mlsanity.com/paper/2102.10929/) (2021)

## Built on by (most cited first, 6 of 6)

- [Segment Any Motion in Videos](https://mlsanity.com/paper/2503.22268/) (2025)
- [Zero-Shot Monocular Motion Segmentation in the Wild by Combining Deep Learning with Geometric Motion Model Fusion](https://mlsanity.com/paper/2405.01723/) (2024)
- [A Unified Model Selection Technique for Spectral Clustering Based Motion Segmentation](https://mlsanity.com/paper/2403.01606/) (2024)
- [Invariant-based Mapping of Space During General Motion of an Observer](https://mlsanity.com/paper/2311.11130/) (2023)
- [Dense Monocular Motion Segmentation Using Optical Flow and Pseudo Depth Map: A Zero-Shot Approach](https://mlsanity.com/paper/2406.18837/) (2024)
- [STAR-VLM: Spatiotemporal Grounding Vision-Language Models for Motion and Velocity Estimation via Automotive Radar Supervision](https://mlsanity.com/paper/2608.01535/) (2026)

## Abstract (arXiv)

Identifying and segmenting moving objects from a moving monocular camera is difficult when there is unknown camera motion, different types of object motions and complex scene structures. To tackle these challenges, we take advantage of two popular branches of monocular motion segmentation approaches: point trajectory based and optical flow based methods, by synergistically fusing these two highly complementary motion cues at object level. By doing this, we are able to model various complex object motions in different scene structures at once, which has not been achieved by existing methods. We first obtain object-specific point trajectories and optical flow mask for each common object in the video, by leveraging the recent foundational models in object recognition, segmentation and tracking. We then construct two robust affinity matrices representing the pairwise object motion affinities throughout the whole video using epipolar geometry and the motion information provided by optical flow. Finally, co-regularized multi-view spectral clustering is used to fuse the two affinity matrices and obtain the final clustering. Our method shows state-of-the-art performance on the KT3DMoSeg dataset, which contains complex motions and scene structures. Being able to identify moving objects allows us to remove them for map building when using visual SLAM or SFM.