# Point Tracking Improves World Action Models

arXiv: https://arxiv.org/abs/2605.23856 · HTML: https://mlsanity.com/paper/2605.23856/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- Placed in Embodied and robot world models, one of 11 directions in World models.
- Builds on 61 of its 82 references that are in the map.
- First posted on arXiv 22 May 2026.

## Placed in

- World models / [Embodied and robot world models](https://mlsanity.com/wm/embodied-and-robot-world-models/)

## Builds on (most cited first, 30 of 61)

- [Denoising Diffusion Implicit Models](https://mlsanity.com/paper/2010.02502/) (2020)
- [SDXL: Improving Latent Diffusion Models for High-Resolution Image Synthesis](https://mlsanity.com/paper/2307.01952/) (2023)
- [OpenVLA: An Open-Source Vision-Language-Action Model](https://mlsanity.com/paper/2406.09246/) (2024)
- [Diffusion Policy: Visuomotor Policy Learning via Action Diffusion](https://mlsanity.com/paper/2303.04137/) (2023)
- [π0: A Vision-Language-Action Flow Model for General Robot Control](https://mlsanity.com/paper/2410.24164/) (2024)
- [RT-1: Robotics Transformer for Real-World Control at Scale](https://mlsanity.com/paper/2212.06817/) (2022)
- [Decoupled Weight Decay Regularization](https://mlsanity.com/paper/1711.05101/) (2017)
- [π0.5: a Vision-Language-Action Model with Open-World Generalization](https://mlsanity.com/paper/2504.16054/) (2025)
- [Learning Fine-Grained Bimanual Manipulation with Low-Cost Hardware](https://mlsanity.com/paper/2304.13705/) (2023)
- [Octo: An Open-Source Generalist Robot Policy](https://mlsanity.com/paper/2405.12213/) (2024)
- [Fine-Tuning Vision-Language-Action Models: Optimizing Speed and Success](https://mlsanity.com/paper/2502.19645/) (2025)
- [Cosmos World Foundation Model Platform for Physical AI](https://mlsanity.com/paper/2501.03575/) (2025)
- [DROID: A Large-Scale In-The-Wild Robot Manipulation Dataset](https://mlsanity.com/paper/2403.12945/) (2024)
- [VoxPoser: Composable 3D Value Maps for Robotic Manipulation with Language Models](https://mlsanity.com/paper/2307.05973/) (2023)
- [R3M: A Universal Visual Representation for Robot Manipulation](https://mlsanity.com/paper/2203.12601/) (2022)
- [FAST: Efficient Action Tokenization for Vision-Language-Action Models](https://mlsanity.com/paper/2501.09747/) (2025)
- [SmolVLA: A Vision-Language-Action Model for Affordable and Efficient Robotics](https://mlsanity.com/paper/2506.01844/) (2025)
- [SpatialVLA: Exploring Spatial Representations for Visual-Language-Action Model](https://mlsanity.com/paper/2501.15830/) (2025)
- [World Action Models are Zero-shot Policies](https://mlsanity.com/paper/2602.15922/) (2026)
- [CoT-VLA: Visual Chain-of-Thought Reasoning for Vision-Language-Action Models](https://mlsanity.com/paper/2503.22020/) (2025)
- [Unleashing Large-Scale Video Generative Pre-training for Visual Robot Manipulation](https://mlsanity.com/paper/2312.13139/) (2023)
- [ReKep: Spatio-Temporal Reasoning of Relational Keypoint Constraints for Robotic Manipulation](https://mlsanity.com/paper/2409.01652/) (2024)
- [GR-2: A Generative Video-Language-Action Model with Web-Scale Knowledge for Robot Manipulation](https://mlsanity.com/paper/2410.06158/) (2024)
- [TAP-Vid: A Benchmark for Tracking Any Point in a Video](https://mlsanity.com/paper/2211.03726/) (2022)
- [Masked Visual Pre-training for Motor Control](https://mlsanity.com/paper/2203.06173/) (2022)
- [Unified World Models: Coupling Video and Action Diffusion for Pretraining on Large Robotic Datasets](https://mlsanity.com/paper/2504.02792/) (2025)
- [Stochastic Variational Video Prediction](https://mlsanity.com/paper/1710.11252/) (2017)
- [Any-point Trajectory Modeling for Policy Learning](https://mlsanity.com/paper/2401.00025/) (2023)
- [DreamVLA: A Vision-Language-Action Model Dreamed with Comprehensive World Knowledge](https://mlsanity.com/paper/2507.04447/) (2025)
- [HybridVLA: Collaborative Diffusion and Autoregression in a Unified Vision-Language-Action Model](https://mlsanity.com/paper/2503.10631/) (2025)

## Built on by (most cited first, 7 of 7)

- [World Action Models: A Survey](https://mlsanity.com/paper/2606.20781/) (2026)
- [Track4Action: Distilling World-Centric 3D Tracker into Vision-Language-Action Policies](https://mlsanity.com/paper/2608.03727/) (2026)
- [FoMoVLA: Bridging Visual Foresight and Motion Guidance for Vision-Language-Action Models](https://mlsanity.com/paper/2607.14739/) (2026)
- [DC-WAM: Dynamic-Centric Visual Supervision and Reasoning for World-Action Models](https://mlsanity.com/paper/2607.25918/) (2026)
- [Modality-Autoregressive World-Action Models](https://mlsanity.com/paper/2609.17524/) (2026)
- [MT-WAM: Reorienting the One-Pass Predictive Representation Toward Action Generation](https://mlsanity.com/paper/2609.21474/) (2026)
- [JAMB: Joint Action-Motion Diffusion for Bimanual Manipulation](https://mlsanity.com/paper/2609.25322/) (2026)

## Abstract (arXiv)

Robot policy learning benefits from world-action models that capture environment dynamics, but pixel-level prediction entangles dynamics with nuisance factors such as lighting and texture, making learned representations vulnerable to task-irrelevant visual variation. We propose JOPAT, a JOint Pixel-And-Track World-Action Model that predicts latent visual observations, 2D point tracks with visibility, and actions in a single denoising diffusion transformer. The key insight is that tracks provide an explicit representation of motion that captures long-horizon dynamics and remains robust under occlusion or partial out-of-frame motion, offering greater utility than modeling pixel appearance alone. On LIBERO and real-world LeRobot tasks, JOPAT improves over pixel-based baselines, with the largest gains on long-horizon tasks involving occlusion, object interaction, and off-screen motion.