# Segment Any Motion in Videos

arXiv: https://arxiv.org/abs/2503.22268 · HTML: https://mlsanity.com/paper/2503.22268/

## Key facts

- Cited by 35 papers in the map (data through 14 Oct 2026).
- Builds on 40 of its 67 references that are in the map.
- First posted on arXiv 28 Mar 2025.

## Builds on (most cited first, 30 of 40)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Mask R-CNN](https://mlsanity.com/paper/1703.06870/) (2017)
- [SAM 2: Segment Anything in Images and Videos](https://mlsanity.com/paper/2408.00714/) (2024)
- [Is Space-Time Attention All You Need for Video Understanding?](https://mlsanity.com/paper/2102.05095/) (2021)
- [Depth Anything: Unleashing the Power of Large-Scale Unlabeled Data](https://mlsanity.com/paper/2401.10891/) (2024)
- [The 2017 DAVIS Challenge on Video Object Segmentation](https://mlsanity.com/paper/1704.00675/) (2017)
- [CoTracker: It is Better to Track Together](https://mlsanity.com/paper/2307.07635/) (2023)
- [HOI4D: A 4D Egocentric Dataset for Category-Level Human-Object Interaction](https://mlsanity.com/paper/2203.01577/) (2022)
- [Shape of Motion: 4D Reconstruction from a Single Video](https://mlsanity.com/paper/2407.13764/) (2024)
- [EmerNeRF: Emergent Spatial-Temporal Scene Decomposition via Self-Supervision](https://mlsanity.com/paper/2311.02077/) (2023)
- [DynamicStereo: Consistent Dynamic Depth from Stereo Videos](https://mlsanity.com/paper/2305.02296/) (2023)
- [Periodic Vibration Gaussian: Dynamic Urban Scene Reconstruction and Real-time Rendering](https://mlsanity.com/paper/2311.18561/) (2023)
- [DynaSLAM: Tracking, Mapping and Inpainting in Dynamic Scenes](https://mlsanity.com/paper/1806.05620/) (2018)
- [BootsTAP: Bootstrapped Training for Tracking-Any-Point](https://mlsanity.com/paper/2402.00847/) (2024)
- [textitS3Gaussian: Self-Supervised Street Gaussians for Autonomous Driving](https://mlsanity.com/paper/2405.20323/) (2024)
- [ParticleSfM: Exploiting Dense Point Trajectories for Localizing Moving Cameras in the Wild](https://mlsanity.com/paper/2207.09137/) (2022)
- [DS-SLAM: A Semantic Visual SLAM towards Dynamic Environments](https://mlsanity.com/paper/1809.08379/) (2018)
- [Full-Duplex Strategy for Video Object Segmentation](https://mlsanity.com/paper/2108.03151/) (2021)
- [FusionSeg: Learning to combine motion and appearance for fully automatic segmention of generic objects in videos](https://mlsanity.com/paper/1701.05384/) (2017)
- [Unsupervised Moving Object Detection via Contextual Information Separation](https://mlsanity.com/paper/1901.03360/) (2019)
- [It's Moving! A Probabilistic Model for Causal Motion Segmentation in Moving Camera Videos](https://mlsanity.com/paper/1604.00136/) (2016)
- [Segmenting Moving Objects via an Object-Centric Layered Representation](https://mlsanity.com/paper/2207.02206/) (2022)
- [Towards Segmenting Anything That Moves](https://mlsanity.com/paper/1902.03715/) (2019)
- [Progressive-X: Efficient, Anytime, Multi-Model Fitting Algorithm](https://mlsanity.com/paper/1906.02290/) (2019)
- [Learning to Segment Moving Objects](https://mlsanity.com/paper/1712.01127/) (2017)
- [EM-driven unsupervised learning for efficient motion segmentation](https://mlsanity.com/paper/2201.02074/) (2022)
- [Moving Object Segmentation: All You Need Is SAM (and Flow)](https://mlsanity.com/paper/2404.12389/) (2024)
- [EpO-Net: Exploiting Geometric Constraints on Dense Trajectories for Motion Saliency](https://mlsanity.com/paper/1909.13258/) (2019)
- [Monocular Instance Motion Segmentation for Autonomous Driving: KITTI InstanceMotSeg Dataset and Multi-task Baseline](https://mlsanity.com/paper/2008.07008/) (2020)
- [Guided Slot Attention for Unsupervised Video Object Segmentation](https://mlsanity.com/paper/2303.08314/) (2023)

## Built on by (most cited first, 30 of 35)

- [ViPE: Video Pose Engine for 3D Geometric Perception](https://mlsanity.com/paper/2508.10934/) (2025)
- [WoW: Towards a World omniscient World model Through Embodied Interaction](https://mlsanity.com/paper/2509.22642/) (2025)
- [MoVieS: Motion-Aware 4D Dynamic View Synthesis in One Second](https://mlsanity.com/paper/2507.10065/) (2025)
- [iTACO: Interactable Digital Twins of Articulated Objects from Casually Captured RGBD Videos](https://mlsanity.com/paper/2506.08334/) (2025)
- [SymphoMotion: Joint Control of Camera Motion and Object Dynamics for Coherent Video Generation](https://mlsanity.com/paper/2604.03723/) (2026)
- [TrackingWorld: World-centric Monocular 3D Tracking of Almost All Pixels](https://mlsanity.com/paper/2512.08358/) (2025)
- [SM4RT: Learning Structured Motion Geometry for 4D Reconstruction](https://mlsanity.com/paper/2607.22534/) (2026)
- [3AM: 3egment Anything with Geometric Consistency in Videos](https://mlsanity.com/paper/2601.08831/) (2026)
- [WildRayZer: Self-supervised Large View Synthesis in Dynamic Environments](https://mlsanity.com/paper/2601.10716/) (2026)
- [CRePE: Curved Ray Expectation Positional Encoding for Unified-Camera-Controlled Video Generation](https://mlsanity.com/paper/2605.12938/) (2026)
- [GeCo: Evaluating Geometric Consistency for Video Generation via Motion and Structure](https://mlsanity.com/paper/2512.22274/) (2025)
- [Segment Anything for Video: A Comprehensive Review of Video Object Segmentation and Tracking from Past to Future](https://mlsanity.com/paper/2507.22792/) (2025)
- [MoRight: Motion Control Done Right](https://mlsanity.com/paper/2604.07348/) (2026)
- [Ctrl-VI: Controllable Video Synthesis via Variational Inference](https://mlsanity.com/paper/2510.07670/) (2025)
- [SAM 2++: Tracking Anything at Any Granularity](https://mlsanity.com/paper/2510.18822/) (2025)
- [CAMotion: A High-Quality Benchmark for Camouflaged Moving Object Detection in the Wild](https://mlsanity.com/paper/2604.08287/) (2026)
- [GMOS: Grounding Moving Object Segmentation in 3D Space and Time](https://mlsanity.com/paper/2605.30352/) (2026)
- [Layer-Aware Video Composition via Split-then-Merge](https://mlsanity.com/paper/2511.20809/) (2025)
- [The Dynamic Prior: Understanding 3D Structures for Casual Dynamic Videos](https://mlsanity.com/paper/2512.05398/) (2025)
- [MOSAIC-GS: Monocular Scene Reconstruction via Advanced Initialization for Complex Dynamic Environments](https://mlsanity.com/paper/2601.05368/) (2026)
- [CamDirector: Towards Long-Term Coherent Video Trajectory Editing](https://mlsanity.com/paper/2603.02256/) (2026)
- [MotionBits: Video Segmentation through Motion-Level Analysis of Rigid Bodies](https://mlsanity.com/paper/2603.06846/) (2026)
- [TRACE: Thermal Recognition Attentive-Framework for CO2 Emissions from Livestock](https://mlsanity.com/paper/2604.09648/) (2026)
- [GenMatter: Perceiving Physical Objects with Generative Matter Models](https://mlsanity.com/paper/2604.22160/) (2026)
- [Motion Cues from Image-based Point Tracking for LiDAR Scene Flow Estimation](https://mlsanity.com/paper/2605.16922/) (2026)
- [RiGS: Rigid-aware 4D Gaussian Splatting from a Single Monocular Video](https://mlsanity.com/paper/2605.23672/) (2026)
- [A Trajectory-Driven Spatio-Temporal Refinement Solution for CVPR 2026 8th UG2+ Challenge Track 3: DOST](https://mlsanity.com/paper/2606.00522/) (2026)
- [SUMO: Segment and Track Any Motion with Nonlinear State Space Models](https://mlsanity.com/paper/2606.29861/) (2026)
- [`Attention-Guided Cross-Temporal Clustering for Self-Supervised Video Object Segmentation](https://mlsanity.com/paper/2607.07230/) (2026)
- [STAR-VLM: Spatiotemporal Grounding Vision-Language Models for Motion and Velocity Estimation via Automotive Radar Supervision](https://mlsanity.com/paper/2608.01535/) (2026)

## Abstract (arXiv)

Moving object segmentation is a crucial task for achieving a high-level understanding of visual scenes and has numerous downstream applications. Humans can effortlessly segment moving objects in videos. Previous work has largely relied on optical flow to provide motion cues; however, this approach often results in imperfect predictions due to challenges such as partial motion, complex deformations, motion blur and background distractions. We propose a novel approach for moving object segmentation that combines long-range trajectory motion cues with DINO-based semantic features and leverages SAM2 for pixel-level mask densification through an iterative prompting strategy. Our model employs Spatio-Temporal Trajectory Attention and Motion-Semantic Decoupled Embedding to prioritize motion while integrating semantic support. Extensive testing on diverse datasets demonstrates state-of-the-art performance, excelling in challenging scenarios and fine-grained segmentation of multiple objects. Our code is available at https://motion-seg.github.io/.