mlsanityData to 7 Oct 2026

HDGT: Heterogeneous Driving Graph Transformer for Multi-Agent Trajectory Prediction via Scene Encoding

arXiv 2205.09753 ↗First posted 30 Apr 2022cs.AI cs.CV cs.LGcited by 109

Not on the map. Why?

Cited by 21 in End-to-end autonomous driving, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Encoding a driving scene into vector representations has been an essential task for autonomous driving that can benefit downstream tasks e.g. trajectory prediction. The driving scene often involves heterogeneous elements such as the different types of objects (agents, lanes, traffic signs) and the semantic relations between objects are rich and diverse. Meanwhile, there also exist relativity across elements, which means that the spatial relation is a relative concept and need be encoded in a ego-centric manner instead of in a global coordinate system. Based on these observations, we propose Heterogeneous Driving Graph Transformer (HDGT), a backbone modelling the driving scene as a heterogeneous graph with different types of nodes and edges. For heterogeneous graph construction, we connect different types of nodes according to diverse semantic relations. For spatial relation encoding, the coordinates of the node as well as its in-edges are in the local node-centric coordinate system. For the aggregation module in the graph neural network (GNN), we adopt the transformer structure in a hierarchical way to fit the heterogeneous nature of inputs. Experimental results show that HDGT achieves state-of-the-art performance for the task of trajectory prediction, on INTERACTION Prediction Challenge and Waymo Open Motion Challenge.

Key facts. Cited by 109 papers in the map (data through 7 Oct 2026). Builds on 54 of its 71 references that are in the map. First posted on arXiv 30 Apr 2022.

Builds on

54 of 71 references.

References · 54

  1. 2017Attention Is All You Needcited by 59,932
  2. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  3. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  4. 2020Language Models are Few-Shot Learnerscited by 32,199
  5. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  6. 2016Semi-Supervised Classification with Graph Convolutional Networkscited by 11,282
  7. 2017Graph Attention Networkscited by 8,854
  8. 2016Layer Normalizationcited by 5,579
  9. 2016PointNet: Deep Learning on Point Sets for 3D Classification and Segmentationcited by 5,223
  10. 2019nuScenes: A multimodal dataset for autonomous drivingcited by 4,485
  11. 2020wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representationscited by 3,853
  12. 2015Fast R-CNNcited by 2,930
38 more
  1. 2017Modeling Relational Data with Graph Convolutional Networkscited by 1,684
  2. 2021Decision Transformer: Reinforcement Learning via Sequence Modelingcited by 1,528
  3. 2018Relational inductive biases, deep learning, and graph networkscited by 1,341
  4. 2021ViLT: Vision-and-Language Transformer Without Convolution or Region Supervisioncited by 1,224
  5. 2021Scaling Vision Transformerscited by 993
  6. 2019Heterogeneous Graph Attention Networkcited by 732
  7. 2019Argoverse: 3D Tracking and Forecasting with Rich Mapscited by 728
  8. 2018Social GAN: Socially Acceptable Trajectories with Generative Adversarial Networkscited by 687
  9. 2020Heterogeneous Graph Transformercited by 581
  10. 2017Understanding the Effective Receptive Field in Deep Convolutional Neural Networkscited by 549
  11. 2021Large Scale Interactive Motion Forecasting for Autonomous Driving : The Waymo Open Motion Datasetcited by 532
  12. 2020VectorNet: Encoding HD Maps and Agent Dynamics from Vectorized Representationcited by 528
  13. 2019MultiPath: Multiple Probabilistic Anchor Trajectory Hypotheses for Behavior Predictioncited by 417
  14. 2018ChauffeurNet: Learning to Drive by Imitating the Best and Synthesizing the Worstcited by 391
  15. 2020Learning Lane Graph Representations for Motion Forecastingcited by 380
  16. 2020TNT: Target-driveN Trajectory Predictioncited by 345
  17. 2020Social-STGCNN: A Social Spatio-Temporal Graph Convolutional Neural Network for Human Trajectory Predictioncited by 324
  18. 2017DESIRE: Distant Future Prediction in Dynamic Scenes with Interacting Agentscited by 310
  19. 2021DenseTNT: End-to-end Trajectory Prediction from Dense Goal Setscited by 292
  20. 2018Multimodal Trajectory Predictions for Autonomous Driving using Deep Convolutional Networkscited by 289
  21. 2021MultiPath++: Efficient Information Fusion and Trajectory Aggregation for Behavior Predictioncited by 246
  22. 2019INTERACTION Dataset: An INTERnational, Adversarial and Cooperative moTION Dataset in Interactive Driving Scenarios with Semantic Mapscited by 231
  23. 2019Multiple Futures Predictioncited by 178
  24. 2021GOHOME: Graph-Oriented Heatmap Output for future Motion Estimationcited by 146
  25. 2019Multi-Agent Tensor Fusion for Contextual Trajectory Predictioncited by 131
  26. 2019Rules of the Road: Predicting Driving Behavior with a Convolutional Model of Semantic Interactionscited by 117
  27. 2020Implicit Latent Variable Model for Scene-Consistent Motion Forecastingcited by 116
  28. 2021LaneRCNN: Distributed Representations for Graph-Centric Motion Forecastingcited by 112
  29. 2021Multimodal Trajectory Prediction Conditioned on Lane-Graph Traversalscited by 112
  30. 2021THOMAS: Trajectory Heatmap Output with learned Multi-Agent Samplingcited by 101
  31. 2021TPCN: Temporal Point Cloud Networks for Motion Forecastingcited by 94
  32. 2021Multi-modal Motion Prediction with Transformer-based Neural Network for Autonomous Drivingcited by 73
  33. 2019An Attention-based Graph Neural Network for Heterogeneous Structural Learningcited by 71
  34. 2020MANTRA: Memory Augmented Networks for Multiple Trajectory Predictioncited by 69
  35. 2020Trajectron++: Dynamically-Feasible Trajectory Forecasting With Heterogeneous Datacited by 58
  36. 2021Knowledge-Preserving Incremental Social Event Detection via Heterogeneous GNNscited by 51
  37. 2020Shared Cross-Modal Trajectory Prediction for Autonomous Drivingcited by 31
  38. 2018Generative Modeling of Multimodal Multi-Human Behaviorcited by 24

50 most cited of 54 All 54 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 7; 2023: 25; 2024: 25; 2025: 35; 2026: 17.

109 papers.

Sort:

By area:

  1. 2022Planning-oriented Autonomous Drivingcited by 1,068
  2. 2022Motion Transformer with Global Intention Localization and Local Movement Refinementcited by 267
  3. 2024Bench2Drive: Towards Multi-Ability Benchmarking of Closed-Loop End-To-End Autonomous Drivingcited by 250
  4. 2022Safety-Enhanced Autonomous Driving Using Interpretable Sensor Fusion Transformercited by 211
  5. 2023MotionLM: Multi-Agent Motion Forecasting as Language Modelingcited by 154
  6. 2023DriveAdapter: Breaking the Coupling Barrier of Perception and Planning in End-to-End Autonomous Drivingcited by 142
  7. 2023GameFormer: Game-theoretic Modeling and Learning of Transformer-based Interactive Prediction and Planning for Autonomous Drivingcited by 141
  8. 2023MTR++: Multi-Agent Motion Prediction with Symmetric Scene Modeling and Guided Intention Queryingcited by 139
  9. 2023LLM4Drive: A Survey of Large Language Models for Autonomous Drivingcited by 136
  10. 2025DriveTransformer: Unified Transformer for Scalable End-to-End Autonomous Drivingcited by 125
  11. 2023ReasonNet: End-to-End Driving with Temporal and Global Reasoningcited by 92
  12. 2025DriveMoE: Mixture-of-Experts for Vision-Language-Action Model in End-to-End Autonomous Drivingcited by 85
38 more
  1. 2023Forecast-MAE: Self-supervised Pre-training for Motion Forecasting with Masked Autoencoderscited by 72
  2. 2022GoRela: Go Relative for Viewpoint-Invariant Motion Forecastingcited by 68
  3. 2025A Survey of World Models for Autonomous Drivingcited by 60
  4. 2023Graph-based Topology Reasoning for Driving Scenescited by 53
  5. 2024BehaviorGPT: Smart Agent Simulation for Autonomous Driving with Next-Patch Predictioncited by 49
  6. 2025Raw2Drive: Reinforcement Learning with Aligned World Models for End-to-End Autonomous Driving (in CARLA v2)cited by 45
  7. 2023Traj-MAE: Masked Autoencoders for Trajectory Predictioncited by 42
  8. 2025ReSim: Reliable World Simulation for Autonomous Drivingcited by 42
  9. 2022FJMP: Factorized Joint Multi-Agent Motion Prediction over Learned Directed Acyclic Interaction Graphscited by 41
  10. 2024HPNet: Dynamic Trajectory Forecasting with Historical Prediction Attentioncited by 37
  11. 2023Real-Time Motion Prediction via Heterogeneous Polyline Transformer with Relative Pose Encodingcited by 32
  12. 2024V2XPnP: Vehicle-to-Everything Spatio-Temporal Fusion for Multi-Agent Perception and Predictioncited by 31
  13. 2024DriveGPT: Scaling Autoregressive Behavior Models for Drivingcited by 29
  14. 2023SEPT: Towards Efficient Scene Representation Learning for Motion Predictioncited by 27
  15. 2024SIMPL: A Simple and Efficient Multi-agent Motion Prediction Baseline for Autonomous Drivingcited by 27
  16. 2024GraphAD: Interaction Scene Graph for End-to-end Autonomous Drivingcited by 25
  17. 2023Open-sourced Data Ecosystem in Autonomous Driving: the Present and Futurecited by 22
  18. 2023Learning Cooperative Trajectory Representations for Motion Forecastingcited by 21
  19. 2024Hybrid-Prediction Integrated Planning for Autonomous Drivingcited by 18
  20. 2022MTR-A: 1st Place Solution for 2022 Waymo Open Dataset Challenge -- Motion Predictioncited by 17
  21. 2023EDA: Evolving and Distinct Anchors for Multimodal Motion Predictioncited by 17
  22. 2024ActiveAD: Planning-Oriented Active Learning for End-to-End Autonomous Drivingcited by 16
  23. 2025Co-MTP: A Cooperative Trajectory Prediction Framework with Multi-Temporal Fusion for Autonomous Drivingcited by 14
  24. 2024AMP: Autoregressive Motion Prediction Revisited with Next Token Prediction for Autonomous Drivingcited by 13
  25. 2025UniMM: A Unified Mixture Model Framework for Multi-Agent Simulationcited by 13
  26. 2024Motion Forecasting in Continuous Drivingcited by 11
  27. 2023Holistic Graph-based Motion Predictioncited by 10
  28. 2023VT-Former: An Exploratory Study on Vehicle Trajectory Prediction for Highway Surveillance through Graph Isomorphism and Transformercited by 10
  29. 2023Categorical Traffic Transformer: Interpretable and Diverse Behavior Prediction with Tokenized Latentcited by 10
  30. 2024SemanticFormer: Holistic and Semantic Traffic Scene Representation for Trajectory Prediction using Knowledge Graphscited by 10
  31. 2024FlatFusion: Delving into Details of Sparse Transformer-based Camera-LiDAR Fusion for Autonomous Drivingcited by 9
  32. 2024Curb Your Attention: Causal Attention Gating for Robust Trajectory Prediction in Autonomous Drivingcited by 9
  33. 2022Exploiting map information for self-supervised learning in motion forecastingcited by 8
  34. 2024MoST: Multi-modality Scene Tokenization for Motion Predictioncited by 8
  35. 2024Anchor3DLane++: 3D Lane Detection via Sample-Adaptive Sparse 3D Anchor Regressioncited by 8
  36. 2025Motion Forecasting for Autonomous Vehicles: A Surveycited by 8
  37. 2024TrACT: A Training Dynamics Aware Contrastive Learning Framework for Long-tail Trajectory Predictioncited by 7
  38. 2025TrajFlow: Multi-modal Motion Prediction via Flow Matchingcited by 7

50 most cited of 109 All 109 (JSON)