# Visual Prompt Guided Unified Pushing Policy

arXiv: https://arxiv.org/abs/2602.19193 · HTML: https://mlsanity.com/paper/2602.19193/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 24 of its 35 references that are in the map.
- First posted on arXiv 22 Feb 2026.

## Builds on (most cited first, 24 of 24)

- [Segment Anything](https://mlsanity.com/paper/2304.02643/) (2023)
- [Scalable Diffusion Models with Transformers](https://mlsanity.com/paper/2212.09748/) (2022)
- [Classifier-Free Diffusion Guidance](https://mlsanity.com/paper/2207.12598/) (2022)
- [Flow Matching for Generative Modeling](https://mlsanity.com/paper/2210.02747/) (2022)
- [Diffusion Policy: Visuomotor Policy Learning via Action Diffusion](https://mlsanity.com/paper/2303.04137/) (2023)
- [Qwen3-VL Technical Report](https://mlsanity.com/paper/2511.21631/) (2025)
- [Goal-Conditioned Imitation Learning using Score-based Diffusion Policies](https://mlsanity.com/paper/2304.02532/) (2023)
- [PIVOT: Iterative Visual Prompting Elicits Actionable Knowledge for VLMs](https://mlsanity.com/paper/2402.07872/) (2024)
- [What Matters in Language Conditioned Robotic Imitation Learning over Unstructured Data](https://mlsanity.com/paper/2204.06252/) (2022)
- [Multimodal Diffusion Transformer: Learning Versatile Behavior from Multimodal Goals](https://mlsanity.com/paper/2407.05996/) (2024)
- [MOKA: Open-World Robotic Manipulation through Mark-Based Visual Prompting](https://mlsanity.com/paper/2403.03174/) (2024)
- [Learning Synergies between Pushing and Grasping with Self-supervised Deep Reinforcement Learning](https://mlsanity.com/paper/1803.09956/) (2018)
- [RT-Sketch: Goal-Conditioned Imitation Learning from Hand-Drawn Sketches](https://mlsanity.com/paper/2403.02709/) (2024)
- [Flow Matching Imitation Learning for Multi-Support Manipulation](https://mlsanity.com/paper/2407.12381/) (2024)
- [Robotic Visual Instruction](https://mlsanity.com/paper/2505.00693/) (2025)
- [A Deep Learning Approach to Grasping the Invisible](https://mlsanity.com/paper/1909.04840/) (2019)
- [Let's Push Things Forward: A Survey on Robot Pushing](https://mlsanity.com/paper/1905.05138/) (2019)
- [Feedback Control of the Pusher-Slider System: A Story of Hybrid and Underactuated Contact Dynamics](https://mlsanity.com/paper/1611.08268/) (2016)
- [Robot Learning of Shifting Objects for Grasping in Cluttered Environments](https://mlsanity.com/paper/1907.11035/) (2019)
- [DIPN: Deep Interaction Prediction Network with Application to Clutter Removal](https://mlsanity.com/paper/2011.04692/) (2020)
- [Learning to Singulate Objects using a Push Proposal Network](https://mlsanity.com/paper/1707.08101/) (2017)
- [Visual Prompting for Robotic Manipulation with Annotation-Guided Pick-and-Place Using ACT](https://mlsanity.com/paper/2508.08748/) (2025)
- [UNO Push: Unified Nonprehensile Object Pushing via Non-Parametric Estimation and Model Predictive Control](https://mlsanity.com/paper/2403.13274/) (2024)
- [Object-Centric Kinodynamic Planning for Nonprehensile Robot Rearrangement Manipulation](https://mlsanity.com/paper/2410.00261/) (2024)

## Abstract (arXiv)

As one of the simplest non-prehensile manipulation skills, pushing has been widely studied as an effective means to rearrange objects. Existing approaches, however, typically rely on multi-step push plans composed of pre-defined pushing primitives with limited application scopes, which restrict their efficiency and versatility across different scenarios. In this work, we propose a unified pushing policy that incorporates a lightweight prompting mechanism into a flow matching policy to guide the generation of reactive, multimodal pushing actions. The visual prompt can be specified by a high-level planner, enabling the reuse of the pushing policy across a wide range of planning problems. Experimental results demonstrate that the proposed unified pushing policy not only outperforms existing baselines but also effectively serves as a low-level primitive within a VLM-guided planning framework to solve table-cleaning tasks efficiently.