mlsanityData to 7 Oct 2026

Rules of the Road: Predicting Driving Behavior with a Convolutional Model of Semantic Interactions

arXiv 1906.08945 ↗First posted 21 Jun 2019cs.CV cs.LG cs.ROcited by 117

Not on the map. Why?

Abstract (arXiv)

We focus on the problem of predicting future states of entities in complex, real-world driving scenarios. Previous research has used low-level signals to predict short time horizons, and has not addressed how to leverage key assets relied upon heavily by industry self-driving systems: (1) large 3D perception efforts which provide highly accurate 3D states of agents with rich attributes, and (2) detailed and accurate semantic maps of the environment (lanes, traffic lights, crosswalks, etc). We present a unified representation which encodes such high-level semantic information in a spatial grid, allowing the use of deep convolutional models to fuse complex scene context. This enables learning entity-entity and entity-environment interactions with simple, feed-forward computations in each timestep within an overall temporal model of an agent's behavior. We propose different ways of modelling the future as a distribution over future states using standard supervised learning. We introduce a novel dataset providing industry-grade rich perception and semantic inputs, and empirically show we can effectively learn fundamentals of driving behavior.

Key facts. Cited by 117 papers in the map (data through 7 Oct 2026). First posted on arXiv 21 Jun 2019.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 28; 2022: 33; 2023: 25; 2024: 16; 2025: 9; 2026: 6.

117 papers.

Sort:

By area:

  1. 2021Large Scale Interactive Motion Forecasting for Autonomous Driving : The Waymo Open Motion Datasetcited by 532
  2. 2021DenseTNT: End-to-end Trajectory Prediction from Dense Goal Setscited by 292
  3. 2022Wayformer: Motion Forecasting via Simple & Efficient Attention Networkscited by 279
  4. 2022Motion Transformer with Global Intention Localization and Local Movement Refinementcited by 267
  5. 2021MultiPath++: Efficient Information Fusion and Trajectory Aggregation for Behavior Predictioncited by 246
  6. 2021FIERY: Future Instance Prediction in Bird's-Eye View from Surround Monocular Camerascited by 239
  7. 2021Multimodal Motion Prediction with Stacked Transformerscited by 207
  8. 2021Scene Transformer: A unified architecture for predicting multiple agent trajectoriescited by 165
  9. 2023MotionLM: Multi-Agent Motion Forecasting as Language Modelingcited by 154
  10. 2023MTR++: Multi-Agent Motion Prediction with Symmetric Scene Modeling and Guided Intention Queryingcited by 139
  11. 2022ViP3D: End-to-end Visual Trajectory Prediction via 3D Agent Queriescited by 116
  12. 2022HDGT: Heterogeneous Driving Graph Transformer for Multi-Agent Trajectory Prediction via Scene Encodingcited by 109
38 more
  1. 2022PlanT: Explainable Planning Transformers via Object-Level Representationscited by 105
  2. 2021THOMAS: Trajectory Heatmap Output with learned Multi-Agent Samplingcited by 101
  3. 2021Structured Bird's-Eye-View Traffic Scene Understanding from Onboard Imagescited by 98
  4. 2021LookOut: Diverse Multi-Future Prediction and Planning for Self-Drivingcited by 96
  5. 2022GoRela: Go Relative for Viewpoint-Invariant Motion Forecastingcited by 68
  6. 2021Learning to Predict Vehicle Trajectories with Model-based Planningcited by 63
  7. 2022Occupancy Flow Fields for Motion Forecasting in Autonomous Drivingcited by 62
  8. 2022Motion Inspired Unsupervised Perception and Prediction in Autonomous Drivingcited by 59
  9. 2022StretchBEV: Stretching Future Instance Prediction Spatially and Temporallycited by 47
  10. 2023Scenario Diffusion: Controllable Driving Scenario Generation With Diffusioncited by 46
  11. 2021Vehicle trajectory prediction works, but not everywherecited by 39
  12. 2023ADAPT: Efficient Multi-Agent Trajectory Prediction with Adaptationcited by 39
  13. 2021Learnable Online Graph Representations for 3D Multi-Object Trackingcited by 38
  14. 2021Divide-and-Conquer for Lane-Aware Diverse Trajectory Predictioncited by 37
  15. 2021Spectral Temporal Graph Neural Network for Trajectory Predictioncited by 36
  16. 2021Spatio-Temporal Graph Dual-Attention Network for Multi-Agent Prediction and Trackingcited by 35
  17. 2024LC-LLM: Explainable Lane-Change Intention and Trajectory Predictions with Large Language Modelscited by 35
  18. 2021Topology Preserving Local Road Network Estimation from Single Onboard Camera Imagecited by 34
  19. 2022Behavioral Intention Prediction in Driving Scenes: A Surveycited by 30
  20. 2023WOMD-LiDAR: Raw Sensor Dataset Benchmark for Motion Forecastingcited by 28
  21. 2023Dynamic Scenario Representation Learning for Motion Forecasting with Heterogeneous Graph Convolutional Recurrent Networkscited by 27
  22. 2023Grid-Centric Traffic Scenario Perception for Autonomous Driving: A Comprehensive Reviewcited by 25
  23. 2023PowerBEV: A Powerful Yet Lightweight Framework for Instance Prediction in Bird's-Eye Viewcited by 24
  24. 2021Learning by Watchingcited by 22
  25. 2021Congestion-aware Multi-agent Trajectory Prediction for Collision Avoidancecited by 21
  26. 2022CRAT-Pred: Vehicle Trajectory Prediction with Crystal Graph Convolutional Neural Networks and Multi-Head Self-Attentioncited by 21
  27. 2022MotionCNN: A Strong Baseline for Motion Prediction in Autonomous Drivingcited by 21
  28. 2021Deep Structured Reactive Planningcited by 18
  29. 2022CausalAgents: A Robustness Benchmark for Motion Forecasting using Causal Relationshipscited by 18
  30. 2023SCENE: Reasoning about Traffic Scenes using Heterogeneous Graph Neural Networkscited by 18
  31. 2021Collaborative Uncertainty in Multi-Agent Trajectory Forecastingcited by 17
  32. 2023Generating Driving Scenes with Diffusioncited by 16
  33. 2021Safety-aware Motion Prediction with Unseen Vehicles for Autonomous Drivingcited by 15
  34. 2022StopNet: Scalable Trajectory and Occupancy Prediction for Urban Autonomous Drivingcited by 14
  35. 2023Scene Informer: Anchor-based Occlusion Inference and Trajectory Prediction in Partially Observable Environmentscited by 14
  36. 2025Trajectory Prediction for Autonomous Driving: Progress, Limitations, and Future Directionscited by 14
  37. 2022Uncertainty estimation for Cross-dataset performance in Trajectory predictioncited by 13
  38. 2022MPA: MultiPath++ Based Architecture for Motion Predictioncited by 13

50 most cited of 117 All 117 (JSON)