# Multimodal Probabilistic Model-Based Planning for Human-Robot Interaction

arXiv: https://arxiv.org/abs/1710.09483 · HTML: https://mlsanity.com/paper/1710.09483/

## Key facts

- Cited by 50 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 25 Oct 2017.

## Built on by (most cited first, 30 of 50)

- [A Survey of Deep RL and IL for Autonomous Driving Policy Learning](https://mlsanity.com/paper/2101.01993/) (2021)
- [The Boltzmann Policy Distribution: Accounting for Systematic Suboptimality in Human Models](https://mlsanity.com/paper/2204.10759/) (2022)
- [Large Language Models as Zero-Shot Human Models for Human-Robot Interaction](https://mlsanity.com/paper/2303.03548/) (2023)
- [The Integration of Prediction and Planning in Deep Learning Automated Driving Systems: A Review](https://mlsanity.com/paper/2308.05731/) (2023)
- [Social Coordination and Altruism in Autonomous Driving](https://mlsanity.com/paper/2107.00200/) (2021)
- [Contingencies from Observations: Tractable Contingency Planning with Learned Behavior Models](https://mlsanity.com/paper/2104.10558/) (2021)
- [Interpretable Self-Aware Neural Networks for Robust Trajectory Prediction](https://mlsanity.com/paper/2211.08701/) (2022)
- [Congestion-aware Multi-agent Trajectory Prediction for Collision Avoidance](https://mlsanity.com/paper/2103.14231/) (2021)
- [Injecting Planning-Awareness into Prediction and Detection Evaluation](https://mlsanity.com/paper/2110.03270/) (2021)
- [Multi-Transmotion: Pre-trained Model for Human Motion Prediction](https://mlsanity.com/paper/2411.02673/) (2024)
- [Deep Structured Reactive Planning](https://mlsanity.com/paper/2101.06832/) (2021)
- [Heterogeneous-Agent Trajectory Forecasting Incorporating Class Uncertainty](https://mlsanity.com/paper/2104.12446/) (2021)
- [Propagating State Uncertainty Through Trajectory Forecasting](https://mlsanity.com/paper/2110.03267/) (2021)
- [Learning Interaction-Aware Trajectory Predictions for Decentralized Multi-Robot Motion Planning in Dynamic Environments](https://mlsanity.com/paper/2102.05382/) (2021)
- [It Takes Two: Learning to Plan for Human-Robot Cooperative Carrying](https://mlsanity.com/paper/2209.12890/) (2022)
- [RAP: Risk-Aware Prediction for Robust Planning](https://mlsanity.com/paper/2210.01368/) (2022)
- [Towards Socially Responsive Autonomous Vehicles: A Reinforcement Learning Framework with Driving Priors and Coordination Awareness](https://mlsanity.com/paper/2309.09726/) (2023)
- [Rethinking Trajectory Forecasting Evaluation](https://mlsanity.com/paper/2107.10297/) (2021)
- [Task-Relevant Failure Detection for Trajectory Predictors in Autonomous Vehicles](https://mlsanity.com/paper/2207.12380/) (2022)
- [On complementing end-to-end human behavior predictors with planning](https://mlsanity.com/paper/2103.05661/) (2021)
- [Multi-Agent Variational Occlusion Inference Using People as Sensors](https://mlsanity.com/paper/2109.02173/) (2021)
- [Learning to Navigate Intersections with Unsupervised Driver Trait Inference](https://mlsanity.com/paper/2109.06783/) (2021)
- [TIP: Task-Informed Motion Prediction for Intelligent Vehicles](https://mlsanity.com/paper/2110.08750/) (2021)
- [Interventional Behavior Prediction: Avoiding Overly Confident Anticipation in Interactive Prediction](https://mlsanity.com/paper/2204.08665/) (2022)
- [Learning Interaction-aware Guidance Policies for Motion Planning in Dense Traffic Scenarios](https://mlsanity.com/paper/2107.04538/) (2021)
- [Receding Horizon Planning with Rule Hierarchies for Autonomous Vehicles](https://mlsanity.com/paper/2212.03323/) (2022)
- [Generative Adversarial Reward Learning for Generalized Behavior Tendency Inference](https://mlsanity.com/paper/2105.00822/) (2021)
- [A Hybrid Rule-Based and Data-Driven Approach to Driver Modeling through Particle Filtering](https://mlsanity.com/paper/2108.12820/) (2021)
- [Personalized Federated Learning of Driver Prediction Models for Autonomous Driving](https://mlsanity.com/paper/2112.00956/) (2021)
- [Quantifying Uncertainty in Motion Prediction with Variational Bayesian Mixture](https://mlsanity.com/paper/2404.03789/) (2024)

## Abstract (arXiv)

This paper presents a method for constructing human-robot interaction policies in settings where multimodality, i.e., the possibility of multiple highly distinct futures, plays a critical role in decision making. We are motivated in this work by the example of traffic weaving, e.g., at highway on-ramps/off-ramps, where entering and exiting cars must swap lanes in a short distance---a challenging negotiation even for experienced drivers due to the inherent multimodal uncertainty of who will pass whom. Our approach is to learn multimodal probability distributions over future human actions from a dataset of human-human exemplars and perform real-time robot policy construction in the resulting environment model through massively parallel sampling of human responses to candidate robot action sequences. Direct learning of these distributions is made possible by recent advances in the theory of conditional variational autoencoders (CVAEs), whereby we learn action distributions simultaneously conditioned on the present interaction history, as well as candidate future robot actions in order to take into account response dynamics. We demonstrate the efficacy of this approach with a human-in-the-loop simulation of a traffic weaving scenario.