# ScePT: Scene-consistent, Policy-based Trajectory Predictions for Planning

arXiv: https://arxiv.org/abs/2206.13387 · HTML: https://mlsanity.com/paper/2206.13387/

## Key facts

- Cited by 64 papers in the map (data through 14 Oct 2026).
- Builds on 31 of its 51 references that are in the map.
- First posted on arXiv 18 Jun 2022.

## Builds on (most cited first, 30 of 31)

- [nuScenes: A multimodal dataset for autonomous driving](https://mlsanity.com/paper/1903.11027/) (2019)
- [Social GAN: Socially Acceptable Trajectories with Generative Adversarial Networks](https://mlsanity.com/paper/1803.10892/) (2018)
- [Attention-Based Models for Speech Recognition](https://mlsanity.com/paper/1506.07503/) (2015)
- [ChauffeurNet: Learning to Drive by Imitating the Best and Synthesizing the Worst](https://mlsanity.com/paper/1812.03079/) (2018)
- [AgentFormer: Agent-Aware Transformers for Socio-Temporal Multi-Agent Forecasting](https://mlsanity.com/paper/2103.14023/) (2021)
- [DESIRE: Distant Future Prediction in Dynamic Scenes with Interacting Agents](https://mlsanity.com/paper/1704.04394/) (2017)
- [SoPhie: An Attentive GAN for Predicting Paths Compliant to Social and Physical Constraints](https://mlsanity.com/paper/1806.01482/) (2018)
- [Multimodal Trajectory Predictions for Autonomous Driving using Deep Convolutional Networks](https://mlsanity.com/paper/1809.10732/) (2018)
- [CoverNet: Multimodal Behavior Prediction using Trajectory Sets](https://mlsanity.com/paper/1911.10298/) (2019)
- [Convolutional Social Pooling for Vehicle Trajectory Prediction](https://mlsanity.com/paper/1805.06771/) (2018)
- [Multimodal Motion Prediction with Stacked Transformers](https://mlsanity.com/paper/2103.11624/) (2021)
- [Social Attention: Modeling Attention in Human Crowds](https://mlsanity.com/paper/1710.04689/) (2017)
- [PRECOG: PREdiction Conditioned On Goals in Visual Multi-Agent Settings](https://mlsanity.com/paper/1905.01296/) (2019)
- [Multiple Futures Prediction](https://mlsanity.com/paper/1911.00997/) (2019)
- [Multi-Agent Tensor Fusion for Contextual Trajectory Prediction](https://mlsanity.com/paper/1904.04776/) (2019)
- [Implicit Latent Variable Model for Scene-Consistent Motion Forecasting](https://mlsanity.com/paper/2007.12036/) (2020)
- [Spatially-Aware Graph Neural Networks for Relational Behavior Forecasting from Sensor Data](https://mlsanity.com/paper/1910.08233/) (2019)
- [LookOut: Diverse Multi-Future Prediction and Planning for Self-Driving](https://mlsanity.com/paper/2101.06547/) (2021)
- [Imitating Driver Behavior with Generative Adversarial Networks](https://mlsanity.com/paper/1701.06699/) (2017)
- [Diverse Trajectory Forecasting with Determinantal Point Processes](https://mlsanity.com/paper/1907.04967/) (2019)
- [Multi-Modal Trajectory Prediction of Surrounding Vehicles with Maneuver based LSTMs](https://mlsanity.com/paper/1805.05499/) (2018)
- [DiversityGAN: Diversity-Aware Vehicle Motion Prediction via Latent Semantic Sampling](https://mlsanity.com/paper/1911.12736/) (2019)
- [CAR-Net: Clairvoyant Attentive Recurrent Network](https://mlsanity.com/paper/1711.10061/) (2017)
- [MultiXNet: Multiclass Multistage Multimodal Motion Prediction](https://mlsanity.com/paper/2006.02000/) (2020)
- [MATS: An Interpretable Trajectory Forecasting Representation for Planning and Control](https://mlsanity.com/paper/2009.07517/) (2020)
- [PredictionNet: Real-Time Joint Probabilistic Traffic Prediction for Planning, Control, and Simulation](https://mlsanity.com/paper/2109.11094/) (2021)
- [The Importance of Prior Knowledge in Precise Multimodal Prediction](https://mlsanity.com/paper/2006.02636/) (2020)
- [Interactive multi-modal motion planning with Branch Model Predictive Control](https://mlsanity.com/paper/2109.05128/) (2021)
- [Grouptron: Dynamic Multi-Scale Graph Convolutional Networks for Group-Aware Dense Crowd Trajectory Forecasting](https://mlsanity.com/paper/2109.14128/) (2021)
- [Reactive motion planning with probabilistic safety guarantees](https://mlsanity.com/paper/2011.03590/) (2020)

## Built on by (most cited first, 30 of 64)

- [Diffusion-based Generation, Optimization, and Planning in 3D Scenes](https://mlsanity.com/paper/2301.06015/) (2023)
- [Guided Conditional Diffusion for Controllable Traffic Simulation](https://mlsanity.com/paper/2210.17366/) (2022)
- [GameFormer: Game-theoretic Modeling and Learning of Transformer-based Interactive Prediction and Planning for Autonomous Driving](https://mlsanity.com/paper/2303.05760/) (2023)
- [Language-Guided Traffic Simulation via Scene-Level Diffusion](https://mlsanity.com/paper/2306.06344/) (2023)
- [LAformer: Trajectory Prediction for Autonomous Driving with Lane-Aware Scene Constraints](https://mlsanity.com/paper/2302.13933/) (2023)
- [Uncovering the Missing Pattern: Unified Framework Towards Trajectory Imputation and Prediction](https://mlsanity.com/paper/2303.16005/) (2023)
- [FJMP: Factorized Joint Multi-Agent Motion Prediction over Learned Directed Acyclic Interaction Graphs](https://mlsanity.com/paper/2211.16197/) (2022)
- [DiffStack: A Differentiable and Modular Control Stack for Autonomous Vehicles](https://mlsanity.com/paper/2212.06437/) (2022)
- [Behavioral Intention Prediction in Driving Scenes: A Survey](https://mlsanity.com/paper/2211.00385/) (2022)
- [The Integration of Prediction and Planning in Deep Learning Automated Driving Systems: A Review](https://mlsanity.com/paper/2308.05731/) (2023)
- [SocialCircle: Learning the Angle-based Social Interaction Representation for Pedestrian Trajectory Prediction](https://mlsanity.com/paper/2310.05370/) (2023)
- [CtRL-Sim: Reactive and Controllable Driving Agents with Offline Reinforcement Learning](https://mlsanity.com/paper/2403.19918/) (2024)
- [GATraj: A Graph- and Attention-based Multi-Agent Trajectory Prediction Model](https://mlsanity.com/paper/2209.07857/) (2022)
- [Reinforcement Learning with Human Feedback for Realistic Traffic Simulation](https://mlsanity.com/paper/2309.00709/) (2023)
- [T4P: Test-Time Training of Trajectory Prediction via Masked Autoencoder and Actor-specific Token Memory](https://mlsanity.com/paper/2403.10052/) (2024)
- [Unsupervised Sampling Promoting for Stochastic Human Trajectory Prediction](https://mlsanity.com/paper/2304.04298/) (2023)
- [Another Vertical View: A Hierarchical Network for Heterogeneous Trajectory Prediction via Spectrums](https://mlsanity.com/paper/2304.05106/) (2023)
- [Trajectory Prediction for Autonomous Driving: Progress, Limitations, and Future Directions](https://mlsanity.com/paper/2503.03262/) (2025)
- [Interactive Joint Planning for Autonomous Vehicles](https://mlsanity.com/paper/2310.18301/) (2023)
- [Intention-aware Denoising Diffusion Model for Trajectory Prediction](https://mlsanity.com/paper/2403.09190/) (2024)
- [Deception Game: Closing the Safety-Learning Loop in Interactive Robot Autonomy](https://mlsanity.com/paper/2309.01267/) (2023)
- [Categorical Traffic Transformer: Interpretable and Diverse Behavior Prediction with Tokenized Latent](https://mlsanity.com/paper/2311.18307/) (2023)
- [Pishgu: Universal Path Prediction Network Architecture for Real-time Cyber-physical Edge Systems](https://mlsanity.com/paper/2210.08057/) (2022)
- [MATRIX: Multi-Agent Trajectory Generation with Diverse Contexts](https://mlsanity.com/paper/2403.06041/) (2024)
- [EgoNav: Egocentric Scene-aware Human Trajectory Prediction](https://mlsanity.com/paper/2403.19026/) (2024)
- [Vision-based Multi-future Trajectory Prediction: A Survey](https://mlsanity.com/paper/2302.10463/) (2023)
- [Multimodal Manoeuvre and Trajectory Prediction for Automated Driving on Highways Using Transformer Networks](https://mlsanity.com/paper/2303.16109/) (2023)
- [Self-supervised Multi-future Occupancy Forecasting for Autonomous Driving](https://mlsanity.com/paper/2407.21126/) (2024)
- [SocialCircle+: Learning the Angle-based Conditioned Interaction Representation for Pedestrian Trajectory Prediction](https://mlsanity.com/paper/2409.14984/) (2024)
- [Trajectory Prediction with Observations of Variable-Length for Motion Planning in Highway Merging scenarios](https://mlsanity.com/paper/2306.05478/) (2023)

## Abstract (arXiv)

Trajectory prediction is a critical functionality of autonomous systems that share environments with uncontrolled agents, one prominent example being self-driving vehicles. Currently, most prediction methods do not enforce scene consistency, i.e., there are a substantial amount of self-collisions between predicted trajectories of different agents in the scene. Moreover, many approaches generate individual trajectory predictions per agent instead of joint trajectory predictions of the whole scene, which makes downstream planning difficult. In this work, we present ScePT, a policy planning-based trajectory prediction model that generates accurate, scene-consistent trajectory predictions suitable for autonomous system motion planning. It explicitly enforces scene consistency and learns an agent interaction policy that can be used for conditional prediction. Experiments on multiple real-world pedestrians and autonomous vehicle datasets show that ScePT} matches current state-of-the-art prediction accuracy with significantly improved scene consistency. We also demonstrate ScePT's ability to work with a downstream contingency planner.