# VIP: Vision Instructed Pre-training for Robotic Manipulation

arXiv: https://arxiv.org/abs/2410.07169 · HTML: https://mlsanity.com/paper/2410.07169/

## Key facts

- Cited by 11 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 9 Oct 2024.

## Built on by (most cited first, 11 of 11)

- [ReconVLA: Reconstructive Vision-Language-Action Model as Effective Robot Perceiver](https://mlsanity.com/paper/2508.10333/) (2025)
- [Point What You Mean: Visually Grounded Instruction Policy](https://mlsanity.com/paper/2512.18933/) (2025)
- [Gaze-Regularized Vision-Language-Action Models for Robotic Manipulation](https://mlsanity.com/paper/2603.23202/) (2026)
- [LiLa-WAM: Lightweight Latent Reasoning World-Action Model for Robotic Manipulation](https://mlsanity.com/paper/2608.03701/) (2026)
- [AnySlot: Goal-Conditioned Vision-Language-Action Policies for Zero-Shot Slot-Level Placement](https://mlsanity.com/paper/2604.10432/) (2026)
- [Lift3D-VLA: Lifting VLA Models to 3D Geometry and Dynamics-Aware Manipulation](https://mlsanity.com/paper/2607.06564/) (2026)
- [Look Where It Matters: Adaptive Visual Refinement for Vision-Language-Action Models](https://mlsanity.com/paper/2608.02197/) (2026)
- [SAM2Grasp: Resolve Multi-modal Grasping via Prompt-conditioned Temporal Action Prediction](https://mlsanity.com/paper/2512.02609/) (2025)
- [SAM3D-Guided Object-Centric Representation Alignment for Vision-Language-Action Models](https://mlsanity.com/paper/2607.25912/) (2026)
- [SurgVLA-Bench: Towards Evaluating Vision-Language-Action Models for Laparoscopic Surgical Robotics](https://mlsanity.com/paper/2606.29247/) (2026)
- [LIBERO-VIFO: Benchmarking the Capability and Safety of Visual Cue Following in Vision-Language-Action Models](https://mlsanity.com/paper/2608.17600/) (2026)

## Abstract (arXiv)

The effectiveness of scaling up training data in robotic manipulation is still limited. A primary challenge in manipulation is the tasks are diverse, and the trained policy would be confused if the task targets are not specified clearly. Existing works primarily rely on text instruction to describe targets. However, we reveal that current robotic data cannot train policies to understand text instruction effectively, and vision is much more comprehensible. Therefore, we introduce utilizing vision instruction to specify targets. A straightforward implementation is training a policy to predict the intermediate actions linking the current observation and a future image. Nevertheless, a single future image does not describe the task target in insufficient detail. To handle this problem, we propose to use sparse point flows to provide more detailed information. Extensive tasks are designed based on real and simulated environments to evaluate the effectiveness of our vision instructed pre-training (VIP) method. The results indicate VIP improves the performance on diverse tasks significantly, and the derived policy can complete competitive tasks like ``opening the lid of a tightly sealed bottle''.