# Where to go next: Learning a Subgoal Recommendation Policy for Navigation Among Pedestrians

arXiv: https://arxiv.org/abs/2102.13073 · HTML: https://mlsanity.com/paper/2102.13073/

## Key facts

- Cited by 28 papers in the map (data through 14 Oct 2026).
- Builds on 20 of its 44 references that are in the map.
- First posted on arXiv 25 Feb 2021.

## Builds on (most cited first, 20 of 20)

- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [High-Dimensional Continuous Control Using Generalized Advantage Estimation](https://mlsanity.com/paper/1506.02438/) (2015)
- [Neural Network Dynamics for Model-Based Deep Reinforcement Learning with Model-Free Fine-Tuning](https://mlsanity.com/paper/1708.02596/) (2017)
- [A Survey of Motion Planning and Control Techniques for Self-driving Urban Vehicles](https://mlsanity.com/paper/1604.07446/) (2016)
- [Crowd-Robot Interaction: Crowd-aware Robot Navigation with Attention-based Deep Reinforcement Learning](https://mlsanity.com/paper/1809.08835/) (2018)
- [Socially Aware Motion Planning with Deep Reinforcement Learning](https://mlsanity.com/paper/1703.08862/) (2017)
- [Decentralized Non-communicating Multiagent Collision Avoidance with Deep Reinforcement Learning](https://mlsanity.com/paper/1609.07845/) (2016)
- [Plan Online, Learn Offline: Efficient Learning and Exploration via Model-Based Control](https://mlsanity.com/paper/1811.01848/) (2018)
- [Exploring Model-based Planning with Policy Networks](https://mlsanity.com/paper/1906.08649/) (2019)
- [Safe Reinforcement Learning Using Robust MPC](https://mlsanity.com/paper/1906.04005/) (2019)
- [Socially Compliant Navigation through Raw Depth Inputs with Generative Adversarial Imitation Learning](https://mlsanity.com/paper/1710.02543/) (2017)
- [Collision Avoidance in Pedestrian-Rich Environments with Deep Reinforcement Learning](https://mlsanity.com/paper/1910.11689/) (2019)
- [Model Predictive Contouring Control for Collision Avoidance in Unstructured Dynamic Environments](https://mlsanity.com/paper/2010.10190/) (2020)
- [Deep Value Model Predictive Control](https://mlsanity.com/paper/1910.03358/) (2019)
- [Practical Reinforcement Learning of Stabilizing Economic MPC](https://mlsanity.com/paper/1904.04614/) (2019)
- [Practical Reinforcement Learning For MPC: Learning from sparse objectives in under an hour on a real robot](https://mlsanity.com/paper/2003.03200/) (2020)
- [Modeling Cooperative Navigation in Dense Human Crowds](https://mlsanity.com/paper/1705.06201/) (2017)
- [Model-based Lookahead Reinforcement Learning](https://mlsanity.com/paper/1908.06012/) (2019)
- [Fully Distributed Multi-Robot Collision Avoidance via Deep Reinforcement Learning for Safe and Efficient Navigation in Complex Scenarios](https://mlsanity.com/paper/1808.03841/) (2018)
- [Combining Benefits from Trajectory Optimization and Deep Reinforcement Learning](https://mlsanity.com/paper/1910.09667/) (2019)

## Built on by (most cited first, 28 of 28)

- [Synthesis of Model Predictive Control and Reinforcement Learning: Survey and Classification](https://mlsanity.com/paper/2502.02133/) (2025)
- [LAMARL: LLM-Aided Multi-Agent Reinforcement Learning for Cooperative Policy Generation](https://mlsanity.com/paper/2506.01538/) (2025)
- [SoNIC: Safe Social Navigation with Adaptive Conformal Inference and Constrained Reinforcement Learning](https://mlsanity.com/paper/2407.17460/) (2024)
- [Handling Sparse Rewards in Reinforcement Learning Using Model Predictive Control](https://mlsanity.com/paper/2210.01525/) (2022)
- [Socially Integrated Navigation: A Social Acting Robot with Deep Reinforcement Learning](https://mlsanity.com/paper/2403.09793/) (2024)
- [A Dynamic Safety Shield for Safe and Efficient Reinforcement Learning of Navigation Tasks](https://mlsanity.com/paper/2412.04153/) (2024)
- [Reinforcement Learning in Robotic Motion Planning by Combined Experience-based Planning and Self-Imitation Learning](https://mlsanity.com/paper/2306.06754/) (2023)
- [Learning-Aided Warmstart of Model Predictive Control in Uncertain Fast-Changing Traffic](https://mlsanity.com/paper/2310.02918/) (2023)
- [Generative AI for Autonomous Driving: A Review](https://mlsanity.com/paper/2505.15863/) (2025)
- [Reinforcement Learning for Robot Navigation with Adaptive Forward Simulation Time (AFST) in a Semi-Markov Model](https://mlsanity.com/paper/2108.06161/) (2021)
- [Holistic Deep-Reinforcement-Learning-based Training of Autonomous Navigation Systems](https://mlsanity.com/paper/2302.02921/) (2023)
- [Disentangling Uncertainty for Safe Social Navigation using Deep Reinforcement Learning](https://mlsanity.com/paper/2409.10655/) (2024)
- [Multi-subgoal Robot Navigation in Crowds with History Information and Interactions](https://mlsanity.com/paper/2205.02003/) (2022)
- [Virtual Guidance as a Mid-level Representation for Navigation with Augmented Reality](https://mlsanity.com/paper/2303.02731/) (2023)
- [Evolutionary Curriculum Training for DRL-Based Navigation Systems](https://mlsanity.com/paper/2306.08870/) (2023)
- [A Reactive Framework for Whole-Body Motion Planning of Mobile Manipulators Combining Reinforcement Learning and SDF-Constrained Quadratic Programmi](https://mlsanity.com/paper/2503.23975/) (2025)
- [Reinforcement Learning with Data Bootstrapping for Dynamic Subgoal Pursuit in Humanoid Robot Navigation](https://mlsanity.com/paper/2506.02206/) (2025)
- [GP-guided MPPI for Efficient Navigation in Complex Unknown Cluttered Environments](https://mlsanity.com/paper/2307.04019/) (2023)
- [Improving the Generalization of Unseen Crowd Behaviors for Reinforcement Learning based Local Motion Planners](https://mlsanity.com/paper/2410.12232/) (2024)
- [A New Trajectory-Oriented Approach to Enhancing Comprehensive Crowd Navigation Performance](https://mlsanity.com/paper/2512.06608/) (2025)
- [KinematicRL: A Sim-to-Real Reinforcement Learning Framework For Social Navigation With Kinodynamic Feasibility](https://mlsanity.com/paper/2606.12042/) (2026)
- [PIER-Flow: Physics-Informed Efficient Rectified Flow for Real-Time Mobile Robot Navigation](https://mlsanity.com/paper/2607.10288/) (2026)
- [Think When It Matters: Conditional VLM Reasoning for Social Navigation with RL Policies](https://mlsanity.com/paper/2607.10991/) (2026)
- [Bridging Reinforcement Learning and Optimal Control via Feasible Action Mapping](https://mlsanity.com/paper/2607.23930/) (2026)
- [Learning-Based Motion Planning for Dynamic Environments: From Foundational Algorithms to Emerging Paradigms](https://mlsanity.com/paper/2608.00625/) (2026)
- [A Hybrid End-to-End and Modular Control Architecture Toward Safe Vehicle Lateral Control: Combining Soft Actor-Critic with Model Predictive Control](https://mlsanity.com/paper/2608.17258/) (2026)
- [SRL-MPC: Shape-Aware Reinforcement Learned Model Predictive Control](https://mlsanity.com/paper/2608.21175/) (2026)
- [RL-Guided PAC-NMPC for Probabilistically-Safe Perception-Based Navigation in Unknown Environments](https://mlsanity.com/paper/2609.39854/) (2026)

## Abstract (arXiv)

Robotic navigation in environments shared with other robots or humans remains challenging because the intentions of the surrounding agents are not directly observable and the environment conditions are continuously changing. Local trajectory optimization methods, such as model predictive control (MPC), can deal with those changes but require global guidance, which is not trivial to obtain in crowded scenarios. This paper proposes to learn, via deep Reinforcement Learning (RL), an interaction-aware policy that provides long-term guidance to the local planner. In particular, in simulations with cooperative and non-cooperative agents, we train a deep network to recommend a subgoal for the MPC planner. The recommended subgoal is expected to help the robot in making progress towards its goal and accounts for the expected interaction with other agents. Based on the recommended subgoal, the MPC planner then optimizes the inputs for the robot satisfying its kinodynamic and collision avoidance constraints. Our approach is shown to substantially improve the navigation performance in terms of number of collisions as compared to prior MPC frameworks, and in terms of both travel time and number of collisions compared to deep RL methods in cooperative, competitive and mixed multiagent scenarios.