# Generalist Robot Manipulation beyond Action Labeled Data

arXiv: https://arxiv.org/abs/2509.19958 · HTML: https://mlsanity.com/paper/2509.19958/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 24 Sep 2025.

## Built on by (most cited first, 6 of 6)

- [AR-VLA: True Autoregressive Action Expert for Vision-Language-Action Models](https://mlsanity.com/paper/2603.10126/) (2026)
- [ActiveMimic: Egocentric Video Pretraining with Active Perception](https://mlsanity.com/paper/2606.06194/) (2026)
- [Robot Learning from Human Videos: A Survey](https://mlsanity.com/paper/2604.27621/) (2026)
- [SPEAR-1: Scaling Beyond Robot Demonstrations via 3D Understanding](https://mlsanity.com/paper/2511.17411/) (2025)
- [Atomic Action Slicing: Planner-Aligned Options for Generalist VLA Agents](https://mlsanity.com/paper/2512.11584/) (2025)
- [Breaking Structural Isolation: Scalable Graph Clustering via Community-Aware Sampling and Structural Entropy](https://mlsanity.com/paper/2607.05469/) (2026)

## Abstract (arXiv)

Recent advances in generalist robot manipulation leverage pre-trained Vision-Language Models (VLMs) and large-scale robot demonstrations to tackle diverse tasks in a zero-shot manner. A key challenge remains: scaling high-quality, action-labeled robot demonstration data, which existing methods rely on for robustness and generalization. To address this, we propose a method that benefits from videos without action labels - featuring humans and/or robots in action - enhancing open-vocabulary performance and enabling data-efficient learning of new tasks. Our method extracts dense, dynamic 3D point clouds at the hand or gripper location and uses a proposed 3D dynamics predictor for self-supervision. This predictor is then tuned to an action predictor using a smaller labeled dataset for action alignment. We show that our method not only learns from unlabeled human and robot demonstrations - improving downstream generalist robot policies - but also enables robots to learn new tasks without action labels (i.e., out-of-action generalization) in both real-world and simulated settings.