# MoSAM: Motion-Guided Segment Anything Model with Spatial-Temporal Memory Selection

arXiv: https://arxiv.org/abs/2505.00739 · HTML: https://mlsanity.com/paper/2505.00739/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Builds on 30 of its 38 references that are in the map.
- First posted on arXiv 30 Apr 2025.

## Builds on (most cited first, 30 of 30)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Segment Anything](https://mlsanity.com/paper/2304.02643/) (2023)
- [Grounded SAM: Assembling Open-World Models for Diverse Visual Tasks](https://mlsanity.com/paper/2401.14159/) (2024)
- [Simple Online and Realtime Tracking](https://mlsanity.com/paper/1602.00763/) (2016)
- [Segment Everything Everywhere All at Once](https://mlsanity.com/paper/2304.06718/) (2023)
- [XMem: Long-Term Video Object Segmentation with an Atkinson-Shiffrin Memory Model](https://mlsanity.com/paper/2207.07115/) (2022)
- [Video Object Segmentation using Space-Time Memory Networks](https://mlsanity.com/paper/1904.00607/) (2019)
- [Video Instance Segmentation](https://mlsanity.com/paper/1905.04804/) (2019)
- [Faster Segment Anything: Towards Lightweight SAM for Mobile Applications](https://mlsanity.com/paper/2306.14289/) (2023)
- [One-Shot Video Object Segmentation](https://mlsanity.com/paper/1611.05198/) (2016)
- [Associating Objects with Transformers for Video Object Segmentation](https://mlsanity.com/paper/2106.02638/) (2021)
- [Putting the Object Back into Video Object Segmentation](https://mlsanity.com/paper/2310.12982/) (2023)
- [Decoupling Features in Hierarchical Propagation for Video Object Segmentation](https://mlsanity.com/paper/2210.09782/) (2022)
- [Collaborative Video Object Segmentation by Foreground-Background Integration](https://mlsanity.com/paper/2003.08333/) (2020)
- [Online Adaptation of Convolutional Neural Networks for Video Object Segmentation](https://mlsanity.com/paper/1706.09364/) (2017)
- [SSTVOS: Sparse Spatiotemporal Transformers for Video Object Segmentation](https://mlsanity.com/paper/2101.08833/) (2021)
- [SAMURAI: Adapting Segment Anything Model for Zero-Shot Visual Tracking with Motion-Aware Memory](https://mlsanity.com/paper/2411.11922/) (2024)
- [VideoMatch: Matching based Video Object Segmentation](https://mlsanity.com/paper/1809.01123/) (2018)
- [Learning What to Learn for Video Object Segmentation](https://mlsanity.com/paper/2003.11540/) (2020)
- [PReMVOS: Proposal-generation, Refinement and Merging for Video Object Segmentation](https://mlsanity.com/paper/1807.09190/) (2018)
- [SAM2Long: Enhancing SAM 2 for Long Video Segmentation with a Training-Free Memory Tree](https://mlsanity.com/paper/2410.16268/) (2024)
- [Blazingly Fast Video Object Segmentation with Pixel-Wise Metric Learning](https://mlsanity.com/paper/1804.03131/) (2018)
- [Joint Inductive and Transductive Learning for Video Object Segmentation](https://mlsanity.com/paper/2108.03679/) (2021)
- [Fast and Accurate Online Video Object Segmentation via Tracking Parts](https://mlsanity.com/paper/1806.02323/) (2018)
- [Recurrent Dynamic Embedding for Video Object Segmentation](https://mlsanity.com/paper/2205.03761/) (2022)
- [Opening up Open-World Tracking](https://mlsanity.com/paper/2104.11221/) (2021)
- [MaskRNN: Instance Level Video Object Segmentation](https://mlsanity.com/paper/1803.11187/) (2018)
- [EmbodiedSAM: Online Segment Any 3D Thing in Real Time](https://mlsanity.com/paper/2408.11811/) (2024)
- [Scalable Video Object Segmentation with Simplified Framework](https://mlsanity.com/paper/2308.09903/) (2023)
- [Unified Embedding Alignment for Open-Vocabulary Video Instance Segmentation](https://mlsanity.com/paper/2407.07427/) (2024)

## Built on by (most cited first, 5 of 5)

- [MOSEv2: A More Challenging Dataset for Video Object Segmentation in Complex Scenes](https://mlsanity.com/paper/2508.05630/) (2025)
- [SAMITE: Position Prompted SAM2 with Calibrated Memory for Visual Object Tracking](https://mlsanity.com/paper/2507.21732/) (2025)
- [3AM: 3egment Anything with Geometric Consistency in Videos](https://mlsanity.com/paper/2601.08831/) (2026)
- [SAM2RL: Towards Reinforcement Learning Memory Control in Segment Anything Model 2](https://mlsanity.com/paper/2507.08548/) (2025)
- [Learning to Reason with Persistent Object States for Video Instance Segmentation](https://mlsanity.com/paper/2609.35539/) (2026)

## Abstract (arXiv)

The recent Segment Anything Model 2 (SAM2) has demonstrated exceptional capabilities in interactive object segmentation for both images and videos. However, as a foundational model on interactive segmentation, SAM2 performs segmentation directly based on mask memory from the past six frames, leading to two significant challenges. Firstly, during inference in videos, objects may disappear since SAM2 relies solely on memory without accounting for object motion information, which limits its long-range object tracking capabilities. Secondly, its memory is constructed from fixed past frames, making it susceptible to challenges associated with object disappearance or occlusion, due to potentially inaccurate segmentation results in memory. To address these problems, we present MoSAM, incorporating two key strategies to integrate object motion cues into the model and establish more reliable feature memory. Firstly, we propose Motion-Guided Prompting (MGP), which represents the object motion in both sparse and dense manners, then injects them into SAM2 through a set of motion-guided prompts. MGP enables the model to adjust its focus towards the direction of motion, thereby enhancing the object tracking capabilities. Furthermore, acknowledging that past segmentation results may be inaccurate, we devise a Spatial-Temporal Memory Selection (ST-MS) mechanism that dynamically identifies frames likely to contain accurate segmentation in both pixel- and frame-level. By eliminating potentially inaccurate mask predictions from memory, we can leverage more reliable memory features to exploit similar regions for improving segmentation results. Extensive experiments on various benchmarks of video object segmentation and video instance segmentation demonstrate that our MoSAM achieves state-of-the-art results compared to other competitors.