mlsanityData to 7 Oct 2026

JFP: Joint Future Prediction with Interactive Multi-Agent Modeling for Autonomous Driving

arXiv 2212.08710 ↗First posted 16 Dec 2022cs.MA cs.LG cs.ROcited by 40

On the map: End-to-end autonomous driving → Imitation-learning end-to-end planners

Abstract (arXiv)

We propose JFP, a Joint Future Prediction model that can learn to generate accurate and consistent multi-agent future trajectories. For this task, many different methods have been proposed to capture social interactions in the encoding part of the model, however, considerably less focus has been placed on representing interactions in the decoder and output stages. As a result, the predicted trajectories are not necessarily consistent with each other, and often result in unrealistic trajectory overlaps. In contrast, we propose an end-to-end trainable model that learns directly the interaction between pairs of agents in a structured, graphical model formulation in order to generate consistent future trajectories. It sets new state-of-the-art results on Waymo Open Motion Dataset (WOMD) for the interactive setting. We also investigate a more complex multi-agent setting for both WOMD and a larger internal dataset, where our approach improves significantly on the trajectory overlap metrics while obtaining on-par or better performance on single-agent trajectory metrics.

Key facts. Cited by 40 papers in the map (data through 7 Oct 2026). Placed in Imitation-learning end-to-end planners, one of 9 directions in End-to-end autonomous driving. Builds on 39 of its 47 references that are in the map. First posted on arXiv 16 Dec 2022.

Builds on

39 of 47 references.

References · 39

  1. 2019nuScenes: A multimodal dataset for autonomous drivingcited by 4,485
  2. 2019Argoverse: 3D Tracking and Forecasting with Rich Mapscited by 728
  3. 2021Large Scale Interactive Motion Forecasting for Autonomous Driving : The Waymo Open Motion Datasetcited by 532
  4. 2020VectorNet: Encoding HD Maps and Agent Dynamics from Vectorized Representationcited by 528
  5. 2019MultiPath: Multiple Probabilistic Anchor Trajectory Hypotheses for Behavior Predictioncited by 417
  6. 2018ChauffeurNet: Learning to Drive by Imitating the Best and Synthesizing the Worstcited by 391
  7. 2020Learning Lane Graph Representations for Motion Forecastingcited by 380
  8. 2019Axial Attention in Multidimensional Transformerscited by 350
  9. 2020TNT: Target-driveN Trajectory Predictioncited by 345
  10. 2017DESIRE: Distant Future Prediction in Dynamic Scenes with Interacting Agentscited by 310
  11. 2021DenseTNT: End-to-end Trajectory Prediction from Dense Goal Setscited by 292
  12. 2018Multimodal Trajectory Predictions for Autonomous Driving using Deep Convolutional Networkscited by 289
27 more
  1. 2022Wayformer: Motion Forecasting via Simple & Efficient Attention Networkscited by 279
  2. 2019CoverNet: Multimodal Behavior Prediction using Trajectory Setscited by 259
  3. 2021MultiPath++: Efficient Information Fusion and Trajectory Aggregation for Behavior Predictioncited by 246
  4. 2021End-to-end Interpretable Neural Motion Plannercited by 240
  5. 2019INTERACTION Dataset: An INTERnational, Adversarial and Cooperative moTION Dataset in Interactive Driving Scenarios with Semantic Mapscited by 231
  6. 2020Fast and Furious: Real Time End-to-End 3D Detection, Tracking and Motion Forecasting with a Single Convolutional Netcited by 228
  7. 2021Multimodal Motion Prediction with Stacked Transformerscited by 207
  8. 2019PRECOG: PREdiction Conditioned On Goals in Visual Multi-Agent Settingscited by 178
  9. 2019Multiple Futures Predictioncited by 178
  10. 2021IntentNet: Learning to Predict Intention from Raw Sensor Datacited by 158
  11. 2021GOHOME: Graph-Oriented Heatmap Output for future Motion Estimationcited by 146
  12. 2020Implicit Latent Variable Model for Scene-Consistent Motion Forecastingcited by 116
  13. 2021LaneRCNN: Distributed Representations for Graph-Centric Motion Forecastingcited by 112
  14. 2019Spatially-Aware Graph Neural Networks for Relational Behavior Forecasting from Sensor Datacited by 104
  15. 2019Multi-Head Attention for Multi-Modal Joint Vehicle Motion Forecastingcited by 87
  16. 2022M2I: From Factored Marginal Trajectory Prediction to Interactive Predictioncited by 81
  17. 2019Diverse Trajectory Forecasting with Determinantal Point Processescited by 80
  18. 2020PiP: Planning-informed Trajectory Prediction for Autonomous Drivingcited by 80
  19. 2020What-If Motion Prediction for Autonomous Drivingcited by 80
  20. 2019The Garden of Forking Paths: Towards Multi-Future Trajectory Predictioncited by 72
  21. 2020MANTRA: Memory Augmented Networks for Multiple Trajectory Predictioncited by 69
  22. 2020DSDNet: Deep Structured self-Driving Networkcited by 61
  23. 2021Identifying Driver Interactions via Conditional Behavior Predictioncited by 48
  24. 2021RAIN: Reinforced Hybrid Attention Inference Network for Motion Forecastingcited by 36
  25. 2021DenseTNT: Waymo Open Dataset Motion Prediction Challenge 1st Place Solutioncited by 14
  26. 2020Multimodal Trajectory Prediction via Topological Invariance for Navigation at Uncontrolled Intersectionscited by 6
  27. 2020Trajformer: Trajectory Prediction with Local Self-Attentive Contexts for Autonomous Drivingcited by 6

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 12; 2024: 12; 2025: 10; 2026: 6.

40 papers.

Sort:

By area:

  1. 2023MotionDiffuser: Controllable Multi-Agent Motion Prediction using Diffusioncited by 210
  2. 2023The Waymo Open Sim Agents Challengecited by 86
  3. 2023Multiverse Transformer: 1st Place Solution for Waymo Open Sim Agents Challenge 2023cited by 25
  4. 2023Large Trajectory Models are Scalable Motion Predictors and Plannerscited by 23
  5. 2024AMP: Autoregressive Motion Prediction Revisited with Next Token Prediction for Autonomous Drivingcited by 13
  6. 2023Categorical Traffic Transformer: Interpretable and Diverse Behavior Prediction with Tokenized Latentcited by 10
  7. 2023Energy-based Potential Games for Joint Motion Forecasting and Controlcited by 9
  8. 2023BiFF: Bi-level Future Fusion with Polyline-based Coordinate for Interactive Trajectory Predictioncited by 8
  9. 2024MoST: Multi-modality Scene Tokenization for Motion Predictioncited by 8
  10. 2025Motion Forecasting for Autonomous Vehicles: A Surveycited by 8
  11. 2024JointMotion: Joint Self-Supervision for Joint Motion Predictioncited by 7
  12. 2023The 2nd Place Solution for 2023 Waymo Open Sim Agents Challengecited by 6
28 more
  1. 2023iPLAN: Intent-Aware Planning in Heterogeneous Traffic via Distributed Multi-Agent Reinforcement Learningcited by 5
  2. 2023On a Connection between Differential Games, Optimal Control, and Energy-based Models for Multi-Agent Interactionscited by 4
  3. 2024SceneMotion: From Agent-Centric Embeddings to Scene-Wide Forecastscited by 4
  4. 2024Motion Forecasting via Model-Based Risk Minimizationcited by 4
  5. 2024Episodic Future Thinking Mechanism for Multi-agent Reinforcement Learningcited by 4
  6. 2024CASPFormer: Trajectory Prediction from BEV Images with Deformable Attentioncited by 3
  7. 2025Generative AI for Autonomous Driving: A Reviewcited by 3
  8. 2026ParkDiffusion++: Ego Intention Conditioned Joint Multi-Agent Trajectory Prediction for Automated Parking using Diffusion Modelscited by 3
  9. 2024MAP-Former: Multi-Agent-Pair Gaussian Joint Predictioncited by 2
  10. 2024Scene-Aware Explainable Multimodal Trajectory Predictioncited by 2
  11. 2025RetroMotion: Retrocausal Motion Forecasting Models are Instructablecited by 2
  12. 2025SceneStreamer: Continuous Scenario Generation as Next Token Group Predictioncited by 2
  13. 2024Model Predictive Simulation Using Structured Graphical Models and Transformerscited by 1
  14. 2025EP-Diffuser: An Efficient Diffusion Model for Traffic Scene Generation and Prediction via Polynomial Representationscited by 1
  15. 2025GenAI-based Multi-Agent Reinforcement Learning towards Distributed Agent Intelligence: A Generative-RL Agent Perspectivecited by 1
  16. 2025Driving is a Game: Combining Planning and Prediction with Bayesian Iterative Best Responsecited by 1
  17. 2026Causality-Aware End-to-End Autonomous Driving via Ego-Centric Joint Scene Modelingcited by 1
  18. 2023NashFormer: Leveraging Local Nash Equilibria for Semantically Diverse Trajectory Predictioncited by 0
  19. 2023Pixel State Value Network for Combined Prediction and Planning in Interactive Environmentscited by 0
  20. 20243CSim: CARLA Corner Case Simulation for Control Assessment in Autonomous Drivingcited by 0
  21. 2024Learning Collective Dynamics of Multi-Agent Systems using Event-based Visioncited by 0
  22. 2025Mode Collapse Happens: Evaluating Critical Interactions in Joint Trajectory Prediction Modelscited by 0
  23. 2025ProgD: Progressive Multi-scale Decoding with Dynamic Graphs for Joint Multi-agent Motion Forecastingcited by 0
  24. 2025BEVTraj: Map-Free End-to-End Trajectory Prediction in Bird's-Eye View with Deformable Attention and Sparse Goal Proposalscited by 0
  25. 2026JACoP: Joint Alignment for Compliant Multi-Agent Predictioncited by 0
  26. 2026Rethinking Training & Inference for Forecasting: Linking Winner-Take-All back to GMMscited by 0
  27. 2026What Probing Reveals about Autonomous Driving: Linking Internal Prediction Errors to Ego Planningcited by 0
  28. 2026A Dynamic Scene Interaction Reasoning Framework for Scene-level Lane-Change Intention and Trajectory Prediction of Multiple Interacting Vehiclescited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2021Scene Transformer: A unified architecture for predicting multiple agent trajectories
  2. 2023MotionLM: Multi-Agent Motion Forecasting as Language Modeling
  3. 2025Future-Aware Interaction Network For Motion Forecasting
  4. 2024Diffusion-Based Environment-Aware Trajectory Prediction
  5. 2024DeMo: Decoupling Motion Forecasting into Directional Intentions and Dynamic States
  6. 2022DiPA: Probabilistic Multi-Modal Interactive Prediction for Autonomous Driving
  7. 2023Waymax: An Accelerated, Data-Driven Simulator for Large-Scale Autonomous Driving Research
  8. 2024Tractable Joint Prediction and Planning over Discrete Behavior Modes for Urban Driving