# 3PoinTr: From Human Videos to Robot Policies with 3D Point-Track Plans

arXiv: https://arxiv.org/abs/2603.08485 · HTML: https://mlsanity.com/paper/2603.08485/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 9 Mar 2026.

## Built on by (most cited first, 7 of 7)

- [EgoWAM: World Action Models Beyond Pixels with In-the-Wild Egocentric Human Data](https://mlsanity.com/paper/2607.08436/) (2026)
- [World Action Models: A Survey](https://mlsanity.com/paper/2606.20781/) (2026)
- [Point Tracking Improves World Action Models](https://mlsanity.com/paper/2605.23856/) (2026)
- [Modality-Autoregressive World-Action Models](https://mlsanity.com/paper/2609.17524/) (2026)
- [PointZero: 3D Point Track Completion for Learning Transferable 3D Dynamics](https://mlsanity.com/paper/2609.19142/) (2026)
- [Hand-Object Interaction in the Age of Large Foundation Models:Reconstruction, Generation, and Embodied Transfer](https://mlsanity.com/paper/2607.28394/) (2026)
- [PointWAM: 3D World Action Modeling for Dexterous Robotic Manipulation](https://mlsanity.com/paper/2610.02840/) (2026)

## Abstract (arXiv)

Human videos could reduce the reliance on expensive robot demonstrations, but extracting task-relevant features remains challenging. We introduce 3PoinTr, a method that provides robot policies with a global 3D point-track plan predicted from human videos. Given an initial point cloud, 3PoinTr predicts dense 3D point tracks spanning the full task and compresses them into a small set of tokens that remain fixed throughout execution. At each policy query, the policy combines these plan tokens with the current observation to predict robot actions. This formulation enables the policy to extract useful motion cues from human videos. We find this formulation transfers exceptionally well, even when human demonstration motions are infeasible for robots. In controlled simulation and real-world evaluations, 3PoinTr achieves an average success rate 25.0 percentage points higher than the strongest behavior-cloning and video-pretraining baselines on real-world tasks, and an improvement of 29.6 percentage points in simulation. Ablations establish the benefits of our method, including predicting non-embodiment 3D tracks once as a global plan, Perceiver-IO compression, and a residual cross-attention mechanism. Our point-track prediction network also outperforms a strong 3D-flow baseline by retaining supervision for temporarily occluded points. Project page: https://adamhung60.github.io/3PoinTr/