# Keypoint-based Diffusion for Robotic Motion Planning on the NICOL Robot

arXiv: https://arxiv.org/abs/2509.04076 · HTML: https://mlsanity.com/paper/2509.04076/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 19 of its 26 references that are in the map.
- First posted on arXiv 4 Sep 2025.

## Builds on (most cited first, 19 of 19)

- [PointNet: Deep Learning on Point Sets for 3D Classification and Segmentation](https://mlsanity.com/paper/1612.00593/) (2016)
- [Diffusion Policy: Visuomotor Policy Learning via Action Diffusion](https://mlsanity.com/paper/2303.04137/) (2023)
- [π0: A Vision-Language-Action Flow Model for General Robot Control](https://mlsanity.com/paper/2410.24164/) (2024)
- [FiLM: Visual Reasoning with a General Conditioning Layer](https://mlsanity.com/paper/1709.07871/) (2017)
- [Octo: An Open-Source Generalist Robot Policy](https://mlsanity.com/paper/2405.12213/) (2024)
- [Motion Planning Diffusion: Learning and Planning of Robot Motions with Diffusion Models](https://mlsanity.com/paper/2308.01557/) (2023)
- [DALL-E-Bot: Introducing Web-Scale Diffusion Models to Robotics](https://mlsanity.com/paper/2210.02438/) (2022)
- [Learning Sampling Distributions for Robot Motion Planning](https://mlsanity.com/paper/1709.05448/) (2017)
- [Motion Planning Networks: Bridging the Gap Between Learning-based and Classical Motion Planners](https://mlsanity.com/paper/1907.06013/) (2019)
- [Motion Policy Networks](https://mlsanity.com/paper/2210.12209/) (2022)
- [Batch Informed Trees (BIT*): Sampling-based Optimal Planning via the Heuristically Guided Search of Implicit Random Geometric Graphs](https://mlsanity.com/paper/1405.5848/) (2014)
- [Adaptively Informed Trees (AIT*): Fast Asymptotically Optimal Path Planning through Adaptive Heuristics](https://mlsanity.com/paper/2002.06599/) (2020)
- [Neural MP: A Generalist Neural Motion Planner](https://mlsanity.com/paper/2409.05864/) (2024)
- [NICOL: A Neuro-inspired Collaborative Semi-humanoid Robot that Bridges Social Interaction and Reliable Manipulation](https://mlsanity.com/paper/2305.08528/) (2023)
- [Graph Neural Networks for Motion Planning](https://mlsanity.com/paper/2006.06248/) (2020)
- [CycleIK: Neuro-inspired Inverse Kinematics](https://mlsanity.com/paper/2307.11554/) (2023)
- [CppFlow: Generative Inverse Kinematics for Efficient and Robust Cartesian Path Planning](https://mlsanity.com/paper/2309.09102/) (2023)
- [Diffusing in Someone Else's Shoes: Robotic Perspective Taking with Diffusion](https://mlsanity.com/paper/2404.07735/) (2024)
- [Robot Motion Planning as Video Prediction: A Spatio-Temporal Neural Network-based Motion Planner](https://mlsanity.com/paper/2208.11287/) (2022)

## Abstract (arXiv)

We propose a novel diffusion-based action model for robotic motion planning. Commonly, established numerical planning approaches are used to solve general motion planning problems, but have significant runtime requirements. By leveraging the power of deep learning, we are able to achieve good results in a much smaller runtime by learning from a dataset generated by these planners. While our initial model uses point cloud embeddings in the input to predict keypoint-based joint sequences in its output, we observed in our ablation study that it remained challenging to condition the network on the point cloud embeddings. We identified some biases in our dataset and refined it, which improved the model's performance. Our model, even without the use of the point cloud encodings, outperforms numerical models by an order of magnitude regarding the runtime, while reaching a success rate of up to 90% of collision free solutions on the test set.