mlsanityData to 14 Oct 2026

Deep Structured Reactive Planning

arXiv 2101.06832 ↗First posted 18 Jan 2021cs.CV cs.AI cs.LGcited by 19

Not on the map. Why?

Abstract (arXiv)

An intelligent agent operating in the real-world must balance achieving its goal with maintaining the safety and comfort of not only itself, but also other participants within the surrounding scene. This requires jointly reasoning about the behavior of other actors while deciding its own actions as these two processes are inherently intertwined - a vehicle will yield to us if we decide to proceed first at the intersection but will proceed first if we decide to yield. However, this is not captured in most self-driving pipelines, where planning follows prediction. In this paper we propose a novel data-driven, reactive planning objective which allows a self-driving vehicle to jointly reason about its own plans as well as how other actors will react to them. We formulate the problem as an energy-based deep structured model that is learned from observational data and encodes both the planning and prediction problems. Through simulations based on both real-world driving and synthetically generated dense traffic, we demonstrate that our reactive model outperforms a non-reactive variant in successfully completing highly complex maneuvers (lane merges/turns in traffic) faster, without trading off collision rate.

Key facts. Cited by 19 papers in the map (data through 14 Oct 2026). Builds on 45 of its 65 references that are in the map. First posted on arXiv 18 Jan 2021.

Builds on

45 of 65 references.

References · 45

  1. 2019nuScenes: A multimodal dataset for autonomous drivingcited by 4,523
  2. 2017CARLA: An Open Urban Driving Simulatorcited by 2,449
  3. 2015Variational Inference with Normalizing Flowscited by 2,170
  4. 2016End to End Learning for Self-Driving Carscited by 1,009
  5. 2018Social GAN: Socially Acceptable Trajectories with Generative Adversarial Networkscited by 690
  6. 2020VectorNet: Encoding HD Maps and Agent Dynamics from Vectorized Representationcited by 530
  7. 2017End-to-end Driving via Conditional Imitation Learningcited by 485
  8. 2019MultiPath: Multiple Probabilistic Anchor Trajectory Hypotheses for Behavior Predictioncited by 420
  9. 2018ChauffeurNet: Learning to Drive by Imitating the Best and Synthesizing the Worstcited by 397
  10. 2020Learning Lane Graph Representations for Motion Forecastingcited by 382
  11. 2019PIXOR: Real-time 3D Object Detection from Point Cloudscited by 358
  12. 2017DESIRE: Distant Future Prediction in Dynamic Scenes with Interacting Agentscited by 310
33 more
  1. 2018Multimodal Trajectory Predictions for Autonomous Driving using Deep Convolutional Networkscited by 289
  2. 2019CoverNet: Multimodal Behavior Prediction using Trajectory Setscited by 259
  3. 2021End-to-end Interpretable Neural Motion Plannercited by 242
  4. 2020Fast and Furious: Real Time End-to-End 3D Detection, Tracking and Motion Forecasting with a Single Convolutional Netcited by 228
  5. 2016Professor Forcing: A New Algorithm for Training Recurrent Networkscited by 216
  6. 2019PRECOG: PREdiction Conditioned On Goals in Visual Multi-Agent Settingscited by 179
  7. 2019Multiple Futures Predictioncited by 178
  8. 2020Perceive, Predict, and Plan: Safe Motion Planning Through Interpretable Semantic Representationscited by 153
  9. 2020PnPNet: End-to-End Perception and Prediction with Tracking in the Loopcited by 144
  10. 2020LiDARsim: Realistic LiDAR Simulation by Leveraging the Real Worldcited by 137
  11. 2019Rules of the Road: Predicting Driving Behavior with a Convolutional Model of Semantic Interactionscited by 117
  12. 2020Implicit Latent Variable Model for Scene-Consistent Motion Forecastingcited by 116
  13. 2019Spatially-Aware Graph Neural Networks for Relational Behavior Forecasting from Sensor Datacited by 104
  14. 2018Baidu Apollo EM Motion Plannercited by 100
  15. 2018Driving Policy Transfer via Modularity and Abstractioncited by 92
  16. 2018Uncertainty-aware Short-term Motion Prediction of Traffic Actors for Autonomous Drivingcited by 90
  17. 2020PiP: Planning-informed Trajectory Prediction for Autonomous Drivingcited by 80
  18. 2018Conditional Affordance Learning for Driving in Urban Environmentscited by 70
  19. 2020DSDNet: Deep Structured self-Driving Networkcited by 62
  20. 2015Structured Prediction Energy Networkscited by 59
  21. 2020End-to-end Contextual Perception and Prediction with Interaction Transformercited by 54
  22. 2019Jointly Learnable Behavior and Trajectory Planning for Self-Driving Vehiclescited by 51
  23. 2017Multimodal Probabilistic Model-Based Planning for Human-Robot Interactioncited by 50
  24. 2018Hierarchical Game-Theoretic Planning for Autonomous Vehiclescited by 48
  25. 2019Driving in Dense Traffic with Model-Free Reinforcement Learningcited by 35
  26. 2017End-to-End Learning for Structured Prediction Energy Networkscited by 33
  27. 2015Fully Connected Deep Structured Networkscited by 24
  28. 2014Learning Deep Structured Modelscited by 21
  29. 2019Cooperation-Aware Reinforcement Learning for Merging in Dense Trafficcited by 20
  30. 2017Towards Full Automated Drive in Urban Environments: A Demonstration in GoMentum Station, Californiacited by 17
  31. 2018Courteous Autonomous Carscited by 14
  32. 2017Neural Variational Inference and Learning in Undirected Graphical Modelscited by 4
  33. 2019Amortized Bethe Free Energy Minimization for Learning MRFscited by 2

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2021: 3; 2022: 5; 2023: 5; 2024: 3; 2025: 1; 2026: 2.

19 papers.

Sort:

By area:

  1. 2022Planning-oriented Autonomous Drivingcited by 1,087
  2. 2021Scene Transformer: A unified architecture for predicting multiple agent trajectoriescited by 165
  3. 2023MotionLM: Multi-Agent Motion Forecasting as Language Modelingcited by 156
  4. 2023GameFormer: Game-theoretic Modeling and Learning of Transformer-based Interactive Prediction and Planning for Autonomous Drivingcited by 141
  5. 2022Differentiable Integrated Motion Prediction and Planning with Learnable Cost Function for Autonomous Drivingcited by 73
  6. 2022Hierarchical Model-Based Imitation Learning for Planning in Autonomous Drivingcited by 44
  7. 2023The Integration of Prediction and Planning in Deep Learning Automated Driving Systems: A Reviewcited by 29
  8. 2022Deep Interactive Motion Prediction and Planning: Playing Games with Motion Prediction Modelscited by 27
  9. 2021UMBRELLA: Uncertainty-Aware Model-Based Offline Reinforcement Learning Leveraging Planningcited by 23
  10. 2021End-to-End Interactive Prediction and Planning with Optical Flow Distillation for Autonomous Drivingcited by 17
  11. 2025Co-MTP: A Cooperative Trajectory Prediction Framework with Multi-Temporal Fusion for Autonomous Drivingcited by 14
  12. 2023Interactive Autonomous Navigation with Internal State Inference and Interactivity Estimationcited by 10
7 more
  1. 2022Interventional Behavior Prediction: Avoiding Overly Confident Anticipation in Interactive Predictioncited by 8
  2. 2024MoST: Multi-modality Scene Tokenization for Motion Predictioncited by 8
  3. 2023Interpretable Motion Planner for Urban Driving via Hierarchical Imitation Learningcited by 4
  4. 2024Autonomous vehicle decision and control through reinforcement learning with traffic flow randomizationcited by 0
  5. 2024Tractable Joint Prediction and Planning over Discrete Behavior Modes for Urban Drivingcited by 0
  6. 2026Chatting about Conditional Trajectory Predictioncited by 0
  7. 2026Safety-Oriented Trajectory Ranking from BEV Perception Outputscited by 0