# Dense Monocular Motion Segmentation Using Optical Flow and Pseudo Depth Map: A Zero-Shot Approach

arXiv: https://arxiv.org/abs/2406.18837 · HTML: https://mlsanity.com/paper/2406.18837/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 24 of its 55 references that are in the map.
- First posted on arXiv 27 Jun 2024.

## Builds on (most cited first, 24 of 24)

- [DINOv2: Learning Robust Visual Features without Supervision](https://mlsanity.com/paper/2304.07193/) (2023)
- [Grounding DINO: Marrying DINO with Grounded Pre-Training for Open-Set Object Detection](https://mlsanity.com/paper/2303.05499/) (2023)
- [RAFT: Recurrent All-Pairs Field Transforms for Optical Flow](https://mlsanity.com/paper/2003.12039/) (2020)
- [The 2017 DAVIS Challenge on Video Object Segmentation](https://mlsanity.com/paper/1704.00675/) (2017)
- [Recognize Anything: A Strong Image Tagging Model](https://mlsanity.com/paper/2306.03514/) (2023)
- [YouTube-VOS: Sequence-to-Sequence Video Object Segmentation](https://mlsanity.com/paper/1809.00461/) (2018)
- [Segment and Track Anything](https://mlsanity.com/paper/2305.06558/) (2023)
- [Decoupling Features in Hierarchical Propagation for Video Object Segmentation](https://mlsanity.com/paper/2210.09782/) (2022)
- [Segment Anything Meets Point Tracking](https://mlsanity.com/paper/2307.01197/) (2023)
- [It's Moving! A Probabilistic Model for Causal Motion Segmentation in Moving Camera Videos](https://mlsanity.com/paper/1604.00136/) (2016)
- [Towards Segmenting Anything That Moves](https://mlsanity.com/paper/1902.03715/) (2019)
- [Video Object Segmentation and Tracking: A Survey](https://mlsanity.com/paper/1904.09172/) (2019)
- [Learning to Segment Rigid Motions from Two Frames](https://mlsanity.com/paper/2101.03694/) (2021)
- [Progressive-X: Efficient, Anytime, Multi-Model Fitting Algorithm](https://mlsanity.com/paper/1906.02290/) (2019)
- [Guess What Moves: Unsupervised Video and Image Segmentation by Anticipating Motion](https://mlsanity.com/paper/2205.07844/) (2022)
- [EM-driven unsupervised learning for efficient motion segmentation](https://mlsanity.com/paper/2201.02074/) (2022)
- [Disentangling Architecture and Training for Optical Flow](https://mlsanity.com/paper/2203.10712/) (2022)
- [EpO-Net: Exploiting Geometric Constraints on Dense Trajectories for Motion Saliency](https://mlsanity.com/paper/1909.13258/) (2019)
- [Monocular Instance Motion Segmentation for Autonomous Driving: KITTI InstanceMotSeg Dataset and Multi-task Baseline](https://mlsanity.com/paper/2008.07008/) (2020)
- [RST-MODNet: Real-time Spatio-temporal Moving Object Detection for Autonomous Driving](https://mlsanity.com/paper/1912.00438/) (2019)
- [Motion Segmentation by Exploiting Complementary Geometric Models](https://mlsanity.com/paper/1804.02142/) (2018)
- [Motion Segmentation from a Moving Monocular Camera](https://mlsanity.com/paper/2309.13772/) (2023)
- [Deep Learning for Robust Motion Segmentation with Non-Static Cameras](https://mlsanity.com/paper/2102.10929/) (2021)
- [A Unified Model Selection Technique for Spectral Clustering Based Motion Segmentation](https://mlsanity.com/paper/2403.01606/) (2024)

## Abstract (arXiv)

Motion segmentation from a single moving camera presents a significant challenge in the field of computer vision. This challenge is compounded by the unknown camera movements and the lack of depth information of the scene. While deep learning has shown impressive capabilities in addressing these issues, supervised models require extensive training on massive annotated datasets, and unsupervised models also require training on large volumes of unannotated data, presenting significant barriers for both. In contrast, traditional methods based on optical flow do not require training data, however, they often fail to capture object-level information, leading to over-segmentation or under-segmentation. In addition, they also struggle in complex scenes with substantial depth variations and non-rigid motion, due to the overreliance of optical flow. To overcome these challenges, we propose an innovative hybrid approach that leverages the advantages of both deep learning methods and traditional optical flow based methods to perform dense motion segmentation without requiring any training. Our method initiates by automatically generating object proposals for each frame using foundation models. These proposals are then clustered into distinct motion groups using both optical flow and relative depth maps as motion cues. The integration of depth maps derived from state-of-the-art monocular depth estimation models significantly enhances the motion cues provided by optical flow, particularly in handling motion parallax issues. Our method is evaluated on the DAVIS-Moving and YTVOS-Moving datasets, and the results demonstrate that our method outperforms the best unsupervised method and closely matches with the state-of-theart supervised methods.