# PTP: Parallelized Tracking and Prediction with Graph Neural Networks and Diversity Sampling

arXiv: https://arxiv.org/abs/2003.07847 · HTML: https://mlsanity.com/paper/2003.07847/

## Key facts

- Cited by 14 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 17 Mar 2020.

## Built on by (most cited first, 14 of 14)

- [AgentFormer: Agent-Aware Transformers for Socio-Temporal Multi-Agent Forecasting](https://mlsanity.com/paper/2103.14023/) (2021)
- [Offboard 3D Object Detection from Point Cloud Sequences](https://mlsanity.com/paper/2103.05073/) (2021)
- [3D Siamese Transformer Network for Single Object Tracking on Point Clouds](https://mlsanity.com/paper/2207.11995/) (2022)
- [TransCenter: Transformers with Dense Representations for Multiple-Object Tracking](https://mlsanity.com/paper/2103.15145/) (2021)
- [Deep Learning on Monocular Object Pose Detection and Tracking: A Comprehensive Overview](https://mlsanity.com/paper/2105.14291/) (2021)
- [TrackMPNN: A Message Passing Graph Neural Architecture for Multi-Object Tracking](https://mlsanity.com/paper/2101.04206/) (2021)
- [Joint Metrics Matter: A Better Standard for Trajectory Forecasting](https://mlsanity.com/paper/2305.06292/) (2023)
- [Joint 3D Object Detection and Tracking Using Spatio-Temporal Representation of Camera Image and LiDAR Point Clouds](https://mlsanity.com/paper/2112.07116/) (2021)
- [MEGA: A Memory-Efficient GNN Accelerator Exploiting Degree-Aware Mixed-Precision Quantization](https://mlsanity.com/paper/2311.09775/) (2023)
- [Optimal Solving of Constrained Path-Planning Problems with Graph Convolutional Networks and Optimized Tree Search](https://mlsanity.com/paper/2108.01036/) (2021)
- [FastGL: A GPU-Efficient Framework for Accelerating Sampling-Based GNN Training at Large Scale](https://mlsanity.com/paper/2409.14939/) (2024)
- [Solving Disjunctive Temporal Networks with Uncertainty under Restricted Time-Based Controllability using Tree Search and Graph Neural Networks](https://mlsanity.com/paper/2203.15030/) (2022)
- [FasterVideo: Efficient Online Joint Object Detection And Tracking](https://mlsanity.com/paper/2204.07394/) (2022)
- [Unified Simulation, Perception, and Generation of Human Behavior](https://mlsanity.com/paper/2204.13678/) (2022)

## Abstract (arXiv)

Multi-object tracking (MOT) and trajectory prediction are two critical components in modern 3D perception systems that require accurate modeling of multi-agent interaction. We hypothesize that it is beneficial to unify both tasks under one framework in order to learn a shared feature representation of agent interaction. Furthermore, instead of performing tracking and prediction sequentially which can propagate errors from tracking to prediction, we propose a parallelized framework to mitigate the issue. Also, our parallel track-forecast framework incorporates two additional novel computational units. First, we use a feature interaction technique by introducing Graph Neural Networks (GNNs) to capture the way in which agents interact with one another. The GNN is able to improve discriminative feature learning for MOT association and provide socially-aware contexts for trajectory prediction. Second, we use a diversity sampling function to improve the quality and diversity of our forecasted trajectories. The learned sampling function is trained to efficiently extract a variety of outcomes from a generative trajectory distribution and helps avoid the problem of generating duplicate trajectory samples. We evaluate on KITTI and nuScenes datasets showing that our method with socially-aware feature learning and diversity sampling achieves new state-of-the-art performance on 3D MOT and trajectory prediction. Project website is: https://www.xinshuoweng.com/projects/PTP