# NeuPAN: Direct Point Robot Navigation with End-to-End Model-based Learning

arXiv: https://arxiv.org/abs/2403.06828 · HTML: https://mlsanity.com/paper/2403.06828/

## Key facts

- Cited by 11 papers in the map (data through 14 Oct 2026).
- Builds on 40 of its 76 references that are in the map.
- First posted on arXiv 11 Mar 2024.

## Builds on (most cited first, 30 of 40)

- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [Convex optimization](https://mlsanity.com/paper/2106.01946/) (2021)
- [CARLA: An Open Urban Driving Simulator](https://mlsanity.com/paper/1711.03938/) (2017)
- [Occupancy Networks: Learning 3D Reconstruction in Function Space](https://mlsanity.com/paper/1812.03828/) (2018)
- [PointPillars: Fast Encoders for Object Detection from Point Clouds](https://mlsanity.com/paper/1812.05784/) (2018)
- [BEVFormer: Learning Bird's-Eye-View Representation from Multi-Camera Images via Spatiotemporal Transformers](https://mlsanity.com/paper/2203.17270/) (2022)
- [Planning-oriented Autonomous Driving](https://mlsanity.com/paper/2212.10156/) (2022)
- [End-to-end Driving via Conditional Imitation Learning](https://mlsanity.com/paper/1710.02410/) (2017)
- [Differentiable Convex Optimization Layers](https://mlsanity.com/paper/1910.12430/) (2019)
- [Mobile ALOHA: Learning Bimanual Mobile Manipulation with Low-Cost Whole-Body Teleoperation](https://mlsanity.com/paper/2401.02117/) (2024)
- [Differentiable MPC for End-to-end Planning and Control](https://mlsanity.com/paper/1810.13400/) (2018)
- [FAST-LIO2: Fast Direct LiDAR-inertial Odometry](https://mlsanity.com/paper/2107.06829/) (2021)
- [Crowd-Robot Interaction: Crowd-aware Robot Navigation with Attention-based Deep Reinforcement Learning](https://mlsanity.com/paper/1809.08835/) (2018)
- [Model-Based Deep Learning](https://mlsanity.com/paper/2012.08405/) (2020)
- [A Survey of End-to-End Driving: Architectures and Training Methods](https://mlsanity.com/paper/2003.06404/) (2020)
- [Decentralized Non-communicating Multiagent Collision Avoidance with Deep Reinforcement Learning](https://mlsanity.com/paper/1609.07845/) (2016)
- [Motion Planning Among Dynamic, Decision-Making Agents with Deep Reinforcement Learning](https://mlsanity.com/paper/1805.01956/) (2018)
- [Information Theoretic Model Predictive Control: Theory and Applications to Autonomous Driving](https://mlsanity.com/paper/1707.02342/) (2017)
- [Motion Planning Networks: Bridging the Gap Between Learning-based and Classical Motion Planners](https://mlsanity.com/paper/1907.06013/) (2019)
- [EGO-Planner: An ESDF-free Gradient-based Local Planner for Quadrotors](https://mlsanity.com/paper/2008.08835/) (2020)
- [Relational Graph Learning for Crowd Navigation](https://mlsanity.com/paper/1909.13165/) (2019)
- [Real-time Neural-MPC: Deep Learning Model Predictive Control for Quadrotors and Agile Robotic Platforms](https://mlsanity.com/paper/2203.07747/) (2022)
- [Long-Range Indoor Navigation with PRM-RL](https://mlsanity.com/paper/1902.09458/) (2019)
- [Optimization-Based Collision Avoidance](https://mlsanity.com/paper/1711.03449/) (2017)
- [RAPTOR: Robust and Perception-aware Trajectory Replanning for Quadrotor Fast Flight](https://mlsanity.com/paper/2007.03465/) (2020)
- [Efficient and Robust LiDAR-Based End-to-End Navigation](https://mlsanity.com/paper/2105.09932/) (2021)
- [EPSILON: An Efficient Planning System for Automated Vehicles in Highly Interactive Environments](https://mlsanity.com/paper/2108.07993/) (2021)
- [Learning Convex Optimization Models](https://mlsanity.com/paper/2006.04248/) (2020)
- [Reinforcement Learned Distributed Multi-Robot Navigation with Reciprocal Velocity Obstacle Shaped Rewards](https://mlsanity.com/paper/2203.10229/) (2022)
- [FASTER: Fast and Safe Trajectory Planner for Navigation in Unknown Environments](https://mlsanity.com/paper/2001.04420/) (2020)

## Built on by (most cited first, 11 of 11)

- [Meta-Memory: Retrieving and Integrating Semantic-Spatial Memories for Robot Spatial Reasoning](https://mlsanity.com/paper/2509.20754/) (2025)
- [Opportunistic Collaborative Planning with Large Vision Model Guided Control and Joint Query-Service Optimization](https://mlsanity.com/paper/2504.18057/) (2025)
- [Agentic Self-Evolutionary Replanning for Embodied Navigation](https://mlsanity.com/paper/2603.02772/) (2026)
- [IR-SIM: A Lightweight Declarative Simulator for Navigation Learning and Benchmarking](https://mlsanity.com/paper/2606.08729/) (2026)
- [Unveiling Uncertainty-Aware Autonomous Cooperative Learning Based Planning Strategy](https://mlsanity.com/paper/2510.11041/) (2025)
- [TRANS: Terrain-aware Reinforcement Learning for Agile Navigation of Quadruped Robots under Social Interactions](https://mlsanity.com/paper/2602.12724/) (2026)
- [NavRL++: A System-Level Framework for Improving Sim-to-Real Transfer in Reinforcement Learning-Based Robot Navigation](https://mlsanity.com/paper/2605.15559/) (2026)
- [ReSeFlow: Rectifying SE(3)-Equivariant Policy Learning Flows](https://mlsanity.com/paper/2509.22695/) (2025)
- [FlashNav: Training Deployable Robot Navigation Policies in Seconds](https://mlsanity.com/paper/2606.15846/) (2026)
- [CrossTracer: Cross-Embodiment Navigation via VLA Model Reasoning and Trace Residuals Adapting](https://mlsanity.com/paper/2608.06688/) (2026)
- [SRL-MPC: Shape-Aware Reinforcement Learned Model Predictive Control](https://mlsanity.com/paper/2608.21175/) (2026)

## Abstract (arXiv)

Navigating a nonholonomic robot in a cluttered, unknown environment requires accurate perception and precise motion control for real-time collision avoidance. This paper presents NeuPAN: a real-time, highly accurate, map-free, easy-to-deploy, and environment-invariant robot motion planner. Leveraging a tightly coupled perception-to-control framework, NeuPAN has two key innovations compared to existing approaches: 1) it directly maps raw point cloud data to a latent distance feature space for collision-free motion generation, avoiding error propagation from the perception to control pipeline; 2) it is interpretable from an end-to-end model-based learning perspective. The crux of NeuPAN is solving an end-to-end mathematical model with numerous point-level constraints using a plug-and-play (PnP) proximal alternating-minimization network (PAN), incorporating neurons in the loop. This allows NeuPAN to generate real-time, physically interpretable motions. It seamlessly integrates data and knowledge engines, and its network parameters can be fine-tuned via backpropagation. We evaluate NeuPAN on a ground mobile robot, a wheel-legged robot, and an autonomous vehicle, in extensive simulated and real-world environments. Results demonstrate that NeuPAN outperforms existing baselines in terms of accuracy, efficiency, robustness, and generalization capabilities across various environments, including the cluttered sandbox, office, corridor, and parking lot. We show that NeuPAN works well in unknown and unstructured environments with arbitrarily shaped objects, transforming impassable paths into passable ones.