# SPGrasp: Spatiotemporal Prompt-driven Grasp Synthesis in Dynamic Scenes

arXiv: https://arxiv.org/abs/2508.20547 · HTML: https://mlsanity.com/paper/2508.20547/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 28 of its 42 references that are in the map.
- First posted on arXiv 28 Aug 2025.

## Builds on (most cited first, 28 of 28)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Segment Anything](https://mlsanity.com/paper/2304.02643/) (2023)
- [SAM 2: Segment Anything in Images and Videos](https://mlsanity.com/paper/2408.00714/) (2024)
- [YOLOv3: An Incremental Improvement](https://mlsanity.com/paper/1804.02767/) (2018)
- [DINO: DETR with Improved DeNoising Anchor Boxes for End-to-End Object Detection](https://mlsanity.com/paper/2203.03605/) (2022)
- [ByteTrack: Multi-Object Tracking by Associating Every Detection Box](https://mlsanity.com/paper/2110.06864/) (2021)
- [Medical SAM Adapter: Adapting Segment Anything Model for Medical Image Segmentation](https://mlsanity.com/paper/2304.12620/) (2023)
- [AnyGrasp: Robust and Efficient Grasp Perception in Spatial and Temporal Domains](https://mlsanity.com/paper/2212.08333/) (2022)
- [Closing the Loop for Robotic Grasping: A Real-time, Generative Grasp Synthesis Approach](https://mlsanity.com/paper/1804.05172/) (2018)
- [STORM: Efficient Stochastic Transformer based World Models for Reinforcement Learning](https://mlsanity.com/paper/2310.09615/) (2023)
- [Jacquard: A Large Scale Dataset for Robotic Grasp Detection](https://mlsanity.com/paper/1803.11469/) (2018)
- [Antipodal Robotic Grasping using Generative Residual Convolutional Neural Network](https://mlsanity.com/paper/1909.04810/) (2019)
- [End-to-end Trainable Deep Neural Network for Robotic Grasp Detection and Semantic Segmentation from RGB](https://mlsanity.com/paper/2107.05287/) (2021)
- [Motion Consistency Model: Accelerating Video Diffusion with Disentangled Motion-Appearance Distillation](https://mlsanity.com/paper/2406.06890/) (2024)
- [Multi-agent Embodied AI: Advances and Future Directions](https://mlsanity.com/paper/2505.05108/) (2025)
- [ROI-based Robotic Grasp Detection for Object Overlapping Scenes](https://mlsanity.com/paper/1808.10313/) (2018)
- [BEHAVIOR Robot Suite: Streamlining Real-World Whole-Body Manipulation for Everyday Household Activities](https://mlsanity.com/paper/2503.05652/) (2025)
- [Exploring Pre-trained Text-to-Video Diffusion Models for Referring Video Object Segmentation](https://mlsanity.com/paper/2403.12042/) (2024)
- [Language-guided Robot Grasping: CLIP-based Referring Grasp Synthesis in Clutter](https://mlsanity.com/paper/2311.05779/) (2023)
- [Object-Independent Human-to-Robot Handovers using Real Time Robotic Vision](https://mlsanity.com/paper/2006.01797/) (2020)
- [Dynamic Grasping with Reachability and Motion Awareness](https://mlsanity.com/paper/2103.10562/) (2021)
- [ManiGaussian++: General Robotic Bimanual Manipulation with Hierarchical Gaussian World Model](https://mlsanity.com/paper/2506.19842/) (2025)
- [Universal Visuo-Tactile Video Understanding for Embodied Interaction](https://mlsanity.com/paper/2505.22566/) (2025)
- [EdgeYOLO: An Edge-Real-Time Object Detector](https://mlsanity.com/paper/2302.07483/) (2023)
- [DGBench: An Open-Source, Reproducible Benchmark for Dynamic Grasping](https://mlsanity.com/paper/2204.13879/) (2022)
- [Instance-wise Grasp Synthesis for Robotic Grasping](https://mlsanity.com/paper/2302.07824/) (2023)
- [Pluralistic Salient Object Detection](https://mlsanity.com/paper/2409.02368/) (2024)
- [Show and Grasp: Few-shot Semantic Segmentation for Robot Grasping through Zero-shot Foundation Models](https://mlsanity.com/paper/2404.12717/) (2024)

## Abstract (arXiv)

Real-time interactive grasp synthesis for dynamic objects remains challenging as existing methods fail to achieve low-latency inference while maintaining promptability. To bridge this gap, we propose SPGrasp (spatiotemporal prompt-driven dynamic grasp synthesis), a novel framework extending segment anything model v2 (SAMv2) for video stream grasp estimation. Our core innovation integrates user prompts with spatiotemporal context, enabling real-time interaction with end-to-end latency as low as 59 ms while ensuring temporal consistency for dynamic objects. In benchmark evaluations, SPGrasp achieves instance-level grasp accuracies of 90.6% on OCID and 93.8% on Jacquard. On the challenging GraspNet-1Billion dataset under continuous tracking, SPGrasp achieves 92.0% accuracy with 73.1 ms per-frame latency, representing a 58.5% reduction compared to the prior state-of-the-art promptable method RoG-SAM while maintaining competitive accuracy. Real-world experiments involving 13 moving objects demonstrate a 94.8% success rate in interactive grasping scenarios. These results confirm SPGrasp effectively resolves the latency-interactivity trade-off in dynamic grasp synthesis.