# Leveraging Neural Network Gradients within Trajectory Optimization for Proactive Human-Robot Interactions

arXiv: https://arxiv.org/abs/2012.01027 · HTML: https://mlsanity.com/paper/2012.01027/

## Key facts

- Cited by 9 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 2 Dec 2020.

## Built on by (most cited first, 9 of 9)

- [DiffStack: A Differentiable and Modular Control Stack for Autonomous Vehicles](https://mlsanity.com/paper/2212.06437/) (2022)
- [Injecting Planning-Awareness into Prediction and Detection Evaluation](https://mlsanity.com/paper/2110.03270/) (2021)
- [Rethinking Trajectory Forecasting Evaluation](https://mlsanity.com/paper/2107.10297/) (2021)
- [Task-Relevant Failure Detection for Trajectory Predictors in Autonomous Vehicles](https://mlsanity.com/paper/2207.12380/) (2022)
- [SICNav-Diffusion: Safe and Interactive Crowd Navigation with Diffusion Trajectory Predictions](https://mlsanity.com/paper/2503.08858/) (2025)
- [TIP: Task-Informed Motion Prediction for Intelligent Vehicles](https://mlsanity.com/paper/2110.08750/) (2021)
- [Learned Tree Search for Long-Horizon Social Robot Navigation in Shared Airspace](https://mlsanity.com/paper/2304.01428/) (2023)
- [A Comparative Study of Human Motion Models in Reinforcement Learning Algorithms for Social Robot Navigation](https://mlsanity.com/paper/2503.15127/) (2025)
- [HumanFlow -- Diffusion-Driven MAV Navigation Among Humans via Tightly-Coupled Motion Tracking, Forecasting, and Control](https://mlsanity.com/paper/2605.25685/) (2026)

## Abstract (arXiv)

To achieve seamless human-robot interactions, robots need to intimately reason about complex interaction dynamics and future human behaviors within their motion planning process. However, there is a disconnect between state-of-the-art neural network-based human behavior models and robot motion planners -- either the behavior models are limited in their consideration of downstream planning or a simplified behavior model is used to ensure tractability of the planning problem. In this work, we present a framework that fuses together the interpretability and flexibility of trajectory optimization (TO) with the predictive power of state-of-the-art human trajectory prediction models. In particular, we leverage gradient information from data-driven prediction models to explicitly reason about human-robot interaction dynamics within a gradient-based TO problem. We demonstrate the efficacy of our approach in a multi-agent scenario whereby a robot is required to safely and efficiently navigate through a crowd of up to ten pedestrians. We compare against a variety of planning methods, and show that by explicitly accounting for interaction dynamics within the planner, our method offers safer and more efficient behaviors, even yielding proactive and nuanced behaviors such as waiting for a pedestrian to pass before moving.