mlsanityData to 7 Oct 2026

VectorNet: Encoding HD Maps and Agent Dynamics from Vectorized Representation

arXiv 2005.04259 ↗First posted 8 May 2020cs.CV cs.LG stat.MLcited by 528

Not on the map. Why?

Cited by 92 in End-to-end autonomous driving, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Behavior prediction in dynamic, multi-agent systems is an important problem in the context of self-driving cars, due to the complex representations and interactions of road components, including moving agents (e.g. pedestrians and vehicles) and road context information (e.g. lanes, traffic lights). This paper introduces VectorNet, a hierarchical graph neural network that first exploits the spatial locality of individual road components represented by vectors and then models the high-order interactions among all components. In contrast to most recent approaches, which render trajectories of moving agents and road context information as bird-eye images and encode them with convolutional neural networks (ConvNets), our approach operates on a vector representation. By operating on the vectorized high definition (HD) maps and agent trajectories, we avoid lossy rendering and computationally intensive ConvNet encoding steps. To further boost VectorNet's capability in learning context features, we propose a novel auxiliary task to recover the randomly masked out map entities and agent trajectories based on their context. We evaluate VectorNet on our in-house behavior prediction benchmark and the recently released Argoverse forecasting dataset. Our method achieves on par or better performance than the competitive rendering approach on both benchmarks while saving over 70% of the model parameters with an order of magnitude reduction in FLOPs. It also outperforms the state of the art on the Argoverse dataset.

Key facts. Cited by 528 papers in the map (data through 7 Oct 2026). First posted on arXiv 8 May 2020.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 57; 2022: 91; 2023: 119; 2024: 108; 2025: 88; 2026: 65.

528 papers.

Sort:

By area:

  1. 2022Planning-oriented Autonomous Drivingcited by 1,068
  2. 2023Argoverse 2: Next Generation Datasets for Self-Driving Perception and Forecastingcited by 701
  3. 2023VAD: Vectorized Scene Representation for Efficient Autonomous Drivingcited by 613
  4. 2021Large Scale Interactive Motion Forecasting for Autonomous Driving : The Waymo Open Motion Datasetcited by 532
  5. 2024DriveVLM: The Convergence of Autonomous Driving and Large Vision-Language Modelscited by 482
  6. 2021DenseTNT: End-to-end Trajectory Prediction from Dense Goal Setscited by 292
  7. 2024VADv2: End-to-End Vectorized Autonomous Driving via Probabilistic Planningcited by 287
  8. 2021HDMapNet: An Online HD Map Construction and Evaluation Frameworkcited by 280
  9. 2022Wayformer: Motion Forecasting via Simple & Efficient Attention Networkscited by 279
  10. 2022Motion Transformer with Global Intention Localization and Local Movement Refinementcited by 267
  11. 2021MultiPath++: Efficient Information Fusion and Trajectory Aggregation for Behavior Predictioncited by 246
  12. 2022VectorMapNet: End-to-end Vectorized HD Map Learningcited by 243
38 more
  1. 2021FIERY: Future Instance Prediction in Bird's-Eye View from Surround Monocular Camerascited by 239
  2. 2023Driving with LLMs: Fusing Object-Level Vector Modality for Explainable Autonomous Drivingcited by 231
  3. 2021Multimodal Motion Prediction with Stacked Transformerscited by 207
  4. 2021Scene Transformer: A unified architecture for predicting multiple agent trajectoriescited by 165
  5. 2021GOHOME: Graph-Oriented Heatmap Output for future Motion Estimationcited by 146
  6. 2023LingoQA: Visual Question Answering for Autonomous Drivingcited by 144
  7. 2023StreamMapNet: Streaming Mapping Network for Vectorized Online HD Map Constructioncited by 143
  8. 2023GameFormer: Game-theoretic Modeling and Learning of Transformer-based Interactive Prediction and Planning for Autonomous Drivingcited by 141
  9. 2022PersFormer: 3D Lane Detection via Perspective Transformer and the OpenLane Benchmarkcited by 140
  10. 2023MTR++: Multi-Agent Motion Prediction with Symmetric Scene Modeling and Guided Intention Queryingcited by 139
  11. 2025Diffusion-Based Planning for Autonomous Driving with Flexible Guidancecited by 138
  12. 2021Generating Useful Accident-Prone Driving Scenarios via a Learned Traffic Priorcited by 136
  13. 2022Learning from All Vehiclescited by 135
  14. 2021Latent Variable Sequential Set Transformers For Joint Multi-Agent Motion Predictioncited by 131
  15. 2022GroupNet: Multiscale Hypergraph Neural Networks for Trajectory Prediction with Relational Reasoningcited by 128
  16. 2021HOME: Heatmap Output for future Motion Estimationcited by 122
  17. 2021Urban Driver: Learning to Drive from Real-world Demonstrations Using Policy Gradientscited by 122
  18. 2022ViP3D: End-to-end Visual Trajectory Prediction via 3D Agent Queriescited by 116
  19. 2021LaneRCNN: Distributed Representations for Graph-Centric Motion Forecastingcited by 112
  20. 2021Multimodal Trajectory Prediction Conditioned on Lane-Graph Traversalscited by 112
  21. 2022HDGT: Heterogeneous Driving Graph Transformer for Multi-Agent Trajectory Prediction via Scene Encodingcited by 109
  22. 2022PlanT: Explainable Planning Transformers via Object-Level Representationscited by 105
  23. 2021THOMAS: Trajectory Heatmap Output with learned Multi-Agent Samplingcited by 101
  24. 2021Shifts: A Dataset of Real Distributional Shift Across Multiple Large-Scale Taskscited by 100
  25. 2023EqMotion: Equivariant Multi-agent Motion Prediction with Invariant Interaction Reasoningcited by 97
  26. 2021TPCN: Temporal Point Cloud Networks for Motion Forecastingcited by 94
  27. 2023ReasonNet: End-to-End Driving with Temporal and Global Reasoningcited by 92
  28. 2022Remember Intentions: Retrospective-Memory-based Trajectory Predictioncited by 86
  29. 2022M2I: From Factored Marginal Trajectory Prediction to Interactive Predictioncited by 81
  30. 2023AntGPT: Can Large Language Models Help Long-term Action Anticipation from Videos?cited by 79
  31. 2023Forecast-MAE: Self-supervised Pre-training for Motion Forecasting with Masked Autoencoderscited by 72
  32. 2022Differentiable Integrated Motion Prediction and Planning with Learnable Cost Function for Autonomous Drivingcited by 71
  33. 2021SafetyNet: Safe planning for real-world self-driving vehicles using machine-learned policiescited by 68
  34. 2022GoRela: Go Relative for Viewpoint-Invariant Motion Forecastingcited by 68
  35. 2023FusionAD: Multi-modality Fusion for Prediction and Planning Tasks of Autonomous Drivingcited by 67
  36. 2021LaPred: Lane-Aware Prediction of Multi-Modal Future Trajectories of Dynamic Agentscited by 64
  37. 2021Learning to Predict Vehicle Trajectories with Model-based Planningcited by 63
  38. 2024NeuroNCAP: Photorealistic Closed-loop Safety Testing for Autonomous Drivingcited by 61

50 most cited of 528 All 528 (JSON)