# Generalizable Coarse-to-Fine Robot Manipulation via Language-Aligned 3D Keypoints

arXiv: https://arxiv.org/abs/2509.23575 · HTML: https://mlsanity.com/paper/2509.23575/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 28 Sep 2025.

## Built on by (most cited first, 3 of 3)

- [Point Tracking Improves World Action Models](https://mlsanity.com/paper/2605.23856/) (2026)
- [OMP: One-step Meanflow Policy with Directional Alignment](https://mlsanity.com/paper/2512.19347/) (2025)
- [Multi-Granularity Language-Guided Imitation Learning via Instruction Decomposition](https://mlsanity.com/paper/2609.37135/) (2026)

## Abstract (arXiv)

Hierarchical coarse-to-fine policy, where a coarse branch predicts a region of interest to guide a fine-grained action predictor, has demonstrated significant potential in robotic 3D manipulation tasks by especially enhancing sample efficiency and enabling more precise manipulation. However, even augmented with pre-trained models, these hierarchical policies still suffer from generalization issues. To enhance generalization to novel instructions and environment variations, we propose Coarse-to-fine Language-Aligned manipulation Policy (CLAP), a framework that integrates three key components: 1) task decomposition, 2) VLM fine-tuning for 3D keypoint prediction, and 3) 3D-aware representation. Through comprehensive experiments in simulation and on a real robot, we demonstrate its superior generalization capability. Specifically, on GemBench, a benchmark designed for evaluating generalization, our approach achieves a 12% higher average success rate than the SOTA method while using only 1/5 of the training trajectories. In real-world experiments, our policy, trained on only 10 demonstrations, successfully generalizes to novel instructions and environments.