# Learning Sampling Dictionaries for Efficient and Generalizable Robot Motion Planning with Transformers

arXiv: https://arxiv.org/abs/2306.00851 · HTML: https://mlsanity.com/paper/2306.00851/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- Builds on 32 of its 47 references that are in the map.
- First posted on arXiv 1 Jun 2023.

## Builds on (most cited first, 30 of 32)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Neural Discrete Representation Learning](https://mlsanity.com/paper/1711.00937/) (2017)
- [PointNet++: Deep Hierarchical Feature Learning on Point Sets in a Metric Space](https://mlsanity.com/paper/1706.02413/) (2017)
- [Decision Transformer: Reinforcement Learning via Sequence Modeling](https://mlsanity.com/paper/2106.01345/) (2021)
- [Generating Diverse High-Fidelity Images with VQ-VAE-2](https://mlsanity.com/paper/1906.00446/) (2019)
- [On Layer Normalization in the Transformer Architecture](https://mlsanity.com/paper/2002.04745/) (2020)
- [Vector-quantized Image Modeling with Improved VQGAN](https://mlsanity.com/paper/2110.04627/) (2021)
- [Offline Reinforcement Learning as One Big Sequence Modeling Problem](https://mlsanity.com/paper/2106.02039/) (2021)
- [A Structured Self-attentive Sentence Embedding](https://mlsanity.com/paper/1703.03130/) (2017)
- [Learning Vision-Guided Quadrupedal Locomotion End-to-End with Cross-Modal Transformers](https://mlsanity.com/paper/2107.03996/) (2021)
- [Motion Planning Networks: Bridging the Gap Between Learning-based and Classical Motion Planners](https://mlsanity.com/paper/1907.06013/) (2019)
- [Informed RRT*: Optimal Sampling-based Path Planning Focused via Direct Sampling of an Admissible Ellipsoidal Heuristic](https://mlsanity.com/paper/1404.2334/) (2014)
- [Robot Motion Planning in Learned Latent Spaces](https://mlsanity.com/paper/1807.10366/) (2018)
- [Batch Informed Trees (BIT*): Sampling-based Optimal Planning via the Heuristically Guided Search of Implicit Random Geometric Graphs](https://mlsanity.com/paper/1405.5848/) (2014)
- [Differentiable Spatial Planning using Transformers](https://mlsanity.com/paper/2112.01010/) (2021)
- [Markerless Camera-to-Robot Pose Estimation via Self-supervised Sim-to-Real Transfer](https://mlsanity.com/paper/2302.14332/) (2023)
- [Bimanual Regrasping for Suture Needles using Reinforcement Learning for Rapid Motion Planning](https://mlsanity.com/paper/2011.04813/) (2020)
- [LEGO: Leveraging Experience in Roadmap Generation for Sampling-Based Planning](https://mlsanity.com/paper/1907.09574/) (2019)
- [Learning to Plan in High Dimensions via Neural Exploration-Exploitation Trees](https://mlsanity.com/paper/1903.00070/) (2019)
- [Learning-Based Proxy Collision Detection for Robot Motion Planning Applications](https://mlsanity.com/paper/1902.08164/) (2019)
- [Reducing Collision Checking for Sampling-Based Motion Planning Using Graph Neural Networks](https://mlsanity.com/paper/2210.08864/) (2022)
- [Learning Sampling Distributions Using Local 3D Workspace Decompositions for Motion Planning in High Dimensions](https://mlsanity.com/paper/2010.15335/) (2020)
- [Motion Planning Transformers: A Motion Planning Framework for Mobile Robots](https://mlsanity.com/paper/2106.02791/) (2021)
- [Potential Functions based Sampling Heuristic For Optimal Path Planning](https://mlsanity.com/paper/1704.00264/) (2017)
- [Dynamically Constrained Motion Planning Networks for Non-Holonomic Robots](https://mlsanity.com/paper/2008.05112/) (2020)
- [MPC-MPNet: Model-Predictive Motion Planning Networks for Fast, Near-Optimal Planning under Kinodynamic Constraints](https://mlsanity.com/paper/2101.06798/) (2021)

## Built on by (most cited first, 8 of 8)

- [Embodied Lifelong Learning for Task and Motion Planning](https://mlsanity.com/paper/2307.06870/) (2023)
- [Bridging the gap between Learning-to-plan, Motion Primitives and Safe Reinforcement Learning](https://mlsanity.com/paper/2408.14063/) (2024)
- [Cascaded Diffusion Models for Neural Motion Planning](https://mlsanity.com/paper/2505.15157/) (2025)
- [Transformer-Enhanced Motion Planner: Attention-Guided Sampling for State-Specific Decision Making](https://mlsanity.com/paper/2404.19403/) (2024)
- [Toward Generalist Neural Motion Planners for Robotic Manipulators: Challenges and Opportunities](https://mlsanity.com/paper/2603.24318/) (2026)
- [Flow Motion Policy: Manipulator Motion Planning with Flow Matching Models](https://mlsanity.com/paper/2604.07084/) (2026)
- [Fast Trajectory Planner with a Reinforcement Learning-based Controller for Robotic Manipulators](https://mlsanity.com/paper/2509.17381/) (2025)
- [RRT*former: Environment-Aware Sampling-Based Motion Planning using Transformer](https://mlsanity.com/paper/2511.15414/) (2025)

## Abstract (arXiv)

Motion planning is integral to robotics applications such as autonomous driving, surgical robots, and industrial manipulators. Existing planning methods lack scalability to higher-dimensional spaces, while recent learning based planners have shown promise in accelerating sampling-based motion planners (SMP) but lack generalizability to out-of-distribution environments. To address this, we present a novel approach, Vector Quantized-Motion Planning Transformers (VQ-MPT) that overcomes the key generalization and scaling drawbacks of previous learning-based methods. VQ-MPT consists of two stages. Stage 1 is a Vector Quantized-Variational AutoEncoder model that learns to represent the planning space using a finite number of sampling distributions, and stage 2 is an Auto-Regressive model that constructs a sampling region for SMPs by selecting from the learned sampling distribution sets. By splitting large planning spaces into discrete sets and selectively choosing the sampling regions, our planner pairs well with out-of-the-box SMPs, generating near-optimal paths faster than without VQ-MPT's aid. It is generalizable in that it can be applied to systems of varying complexities, from 2D planar to 14D bi-manual robots with diverse environment representations, including costmaps and point clouds. Trained VQ-MPT models generalize to environments unseen during training and achieve higher success rates than previous methods.