mlsanityData to 14 Oct 2026

MotionLM: Multi-Agent Motion Forecasting as Language Modeling

arXiv 2309.16534 ↗First posted 28 Sep 2023cs.CV cs.AI cs.LGcited by 156

On the map: LLM agents and tool use → Multimodal and embodied agents

Abstract (arXiv)

Reliable forecasting of the future behavior of road agents is a critical component to safe planning in autonomous vehicles. Here, we represent continuous trajectories as sequences of discrete motion tokens and cast multi-agent motion prediction as a language modeling task over this domain. Our model, MotionLM, provides several advantages: First, it does not require anchors or explicit latent variable optimization to learn multimodal distributions. Instead, we leverage a single standard language modeling objective, maximizing the average log probability over sequence tokens. Second, our approach bypasses post-hoc interaction heuristics where individual agent trajectory generation is conducted prior to interactive scoring. Instead, MotionLM produces joint distributions over interactive agent futures in a single autoregressive decoding process. In addition, the model's sequential factorization enables temporally causal conditional rollouts. The proposed approach establishes new state-of-the-art performance for multi-agent motion prediction on the Waymo Open Motion Dataset, ranking 1st on the interactive challenge leaderboard.

Key facts. Cited by 156 papers in the map (data through 14 Oct 2026). Placed in Multimodal and embodied agents, one of 16 directions in LLM agents and tool use. Builds on 46 of its 53 references that are in the map. First posted on arXiv 28 Sep 2023.

Builds on

46 of 53 references.

References · 46

  1. 2017Attention Is All You Needcited by 60,361
  2. 2020Language Models are Few-Shot Learnerscited by 32,419
  3. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,326
  4. 2019nuScenes: A multimodal dataset for autonomous drivingcited by 4,523
  5. 2021Zero-Shot Text-to-Image Generationcited by 4,288
  6. 2019The Curious Case of Neural Text Degenerationcited by 2,916
  7. 2016WaveNet: A Generative Model for Raw Audiocited by 1,722
  8. 2022Scaling Autoregressive Models for Content-Rich Text-to-Image Generationcited by 1,295
  9. 2021Perceiver: General Perception with Iterative Attentioncited by 1,104
  10. 2016Pixel Recurrent Neural Networkscited by 816
  11. 2019Argoverse: 3D Tracking and Forecasting with Rich Mapscited by 732
  12. 2022AudioLM: a Language Modeling Approach to Audio Generationcited by 616
34 more
  1. 2021Large Scale Interactive Motion Forecasting for Autonomous Driving : The Waymo Open Motion Datasetcited by 537
  2. 2019MultiPath: Multiple Probabilistic Anchor Trajectory Hypotheses for Behavior Predictioncited by 420
  3. 2020Learning Lane Graph Representations for Motion Forecastingcited by 382
  4. 2020TNT: Target-driveN Trajectory Predictioncited by 347
  5. 2021AgentFormer: Agent-Aware Transformers for Socio-Temporal Multi-Agent Forecastingcited by 331
  6. 2017DESIRE: Distant Future Prediction in Dynamic Scenes with Interacting Agentscited by 310
  7. 2018Multimodal Trajectory Predictions for Autonomous Driving using Deep Convolutional Networkscited by 289
  8. 2022Wayformer: Motion Forecasting via Simple & Efficient Attention Networkscited by 280
  9. 2022Motion Transformer with Global Intention Localization and Local Movement Refinementcited by 270
  10. 2020PolyGen: An Autoregressive Generative Model of 3D Meshescited by 269
  11. 2021MultiPath++: Efficient Information Fusion and Trajectory Aggregation for Behavior Predictioncited by 246
  12. 2019INTERACTION Dataset: An INTERnational, Adversarial and Cooperative moTION Dataset in Interactive Driving Scenarios with Semantic Mapscited by 231
  13. 2019PRECOG: PREdiction Conditioned On Goals in Visual Multi-Agent Settingscited by 179
  14. 2019Multiple Futures Predictioncited by 178
  15. 2021IntentNet: Learning to Predict Intention from Raw Sensor Datacited by 158
  16. 2021Latent Variable Sequential Set Transformers For Joint Multi-Agent Motion Predictioncited by 131
  17. 2019Rules of the Road: Predicting Driving Behavior with a Convolutional Model of Semantic Interactionscited by 117
  18. 2020Implicit Latent Variable Model for Scene-Consistent Motion Forecastingcited by 116
  19. 2022HDGT: Heterogeneous Driving Graph Transformer for Multi-Agent Trajectory Prediction via Scene Encodingcited by 110
  20. 2019Spatially-Aware Graph Neural Networks for Relational Behavior Forecasting from Sensor Datacited by 104
  21. 2021LookOut: Diverse Multi-Future Prediction and Planning for Self-Drivingcited by 96
  22. 2022M2I: From Factored Marginal Trajectory Prediction to Interactive Predictioncited by 81
  23. 2020PiP: Planning-informed Trajectory Prediction for Autonomous Drivingcited by 80
  24. 2020What-If Motion Prediction for Autonomous Drivingcited by 80
  25. 2021Shaking the foundations: delusions in sequence models for interaction and controlcited by 62
  26. 2022Symphony: Learning Realistic and Diverse Agents for Autonomous Driving Simulationcited by 61
  27. 2020Trajectron++: Dynamically-Feasible Trajectory Forecasting With Heterogeneous Datacited by 58
  28. 2021Identifying Driver Interactions via Conditional Behavior Predictioncited by 48
  29. 2021Deep Structured Reactive Planningcited by 19
  30. 2022LatentFormer: Multi-Agent Transformer-Based Interaction Modeling and Trajectory Predictioncited by 19
  31. 2022MPA: MultiPath++ Based Architecture for Motion Predictioncited by 13
  32. 2022KEMP: Keyframe-Based Hierarchical End-to-End Deep Model for Long-Term Trajectory Predictioncited by 11
  33. 2022Modeling road user response timing in naturalistic settings: a surprise-based frameworkcited by 4
  34. 2022Dense-TNT: Efficient Vehicle Type Classification Neural Network Using Satellite Imagerycited by 3

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 6; 2024: 50; 2025: 65; 2026: 35.

156 papers.

Sort:

By area:

  1. 2023DriveLM: Driving with Graph Visual Question Answeringcited by 536
  2. 2023A Survey on Multimodal Large Language Models for Autonomous Drivingcited by 290
  3. 2024VADv2: End-to-End Vectorized Autonomous Driving via Probabilistic Planningcited by 289
  4. 2024EMMA: End-to-End Multimodal Model for Autonomous Drivingcited by 231
  5. 2024Embodied Understanding of Driving Scenarioscited by 63
  6. 2024Tokenize the World into Object-level Knowledge to Address Long-tail Events in Autonomous Drivingcited by 56
  7. 2024UniTraj: A Unified Framework for Scalable Vehicle Trajectory Predictioncited by 52
  8. 2025From Foresight to Forethought: VLM-In-the-Loop Policy Steering via Latent Alignmentcited by 52
  9. 2024Comp4D: LLM-Guided Compositional 4D Scene Generationcited by 50
  10. 2024BehaviorGPT: Smart Agent Simulation for Autonomous Driving with Next-Patch Predictioncited by 50
  11. 2024SceneDiffuser: Efficient and Controllable Driving Simulation Initialization and Rolloutcited by 48
  12. 2024Diffusion-ES: Gradient-free Planning with Diffusion for Autonomous Driving and Zero-Shot Instruction Followingcited by 42
38 more
  1. 2024Gen-Drive: Enhancing Diffusion Generative Driving Policies with Reward Modeling and Reinforcement Learning Fine-tuningcited by 42
  2. 2024Solving Motion Planning Tasks with a Scalable Generative Modelcited by 40
  3. 2024Closed-Loop Supervised Fine-Tuning of Tokenized Traffic Modelscited by 40
  4. 2024HPNet: Dynamic Trajectory Forecasting with Historical Prediction Attentioncited by 38
  5. 2024LC-LLM: Explainable Lane-Change Intention and Trajectory Predictions with Large Language Modelscited by 35
  6. 2025CarPlanner: Consistent Auto-regressive Trajectory Planning for Large-scale Reinforcement Learning in Autonomous Drivingcited by 34
  7. 2024DriveGPT: Scaling Autoregressive Behavior Models for Drivingcited by 29
  8. 2024CtRL-Sim: Reactive and Controllable Driving Agents with Offline Reinforcement Learningcited by 27
  9. 2024Improving Agent Behaviors with RL Fine-tuning for Autonomous Drivingcited by 27
  10. 2024SMART: Scalable Multi-agent Real-time Motion Generation via Next-token Predictioncited by 25
  11. 2023Large Trajectory Models are Scalable Motion Predictors and Plannerscited by 23
  12. 2024Adapting to Length Shift: FlexiLength Network for Trajectory Predictioncited by 22
  13. 2024Multi-Agent Dynamic Relational Reasoning for Social Robot Navigationcited by 20
  14. 2025Plan-R1: Safe and Feasible Trajectory Planning as Language Modelingcited by 20
  15. 2024Sports-Traj: A Unified Trajectory Generation Model for Multi-Agent Movement in Sportscited by 19
  16. 2024WOMD-Reasoning: A Large-Scale Dataset for Interaction Reasoning in Drivingcited by 19
  17. 2025Embodied Scene Understanding for Vision Language Models via MetaVQAcited by 19
  18. 2025S4-Driver: Scalable Self-Supervised Driving Multimodal Large Language Modelwith Spatio-Temporal Visual Representationcited by 19
  19. 2025Scaling Laws of Motion Forecasting and Planning -- Technical Reportcited by 19
  20. 2024KiGRAS: Kinematic-Driven Generative Model for Realistic Agent Simulationcited by 17
  21. 2024ChatCam: Empowering Camera Control through Conversational AIcited by 15
  22. 2025Trajectory Prediction Meets Large Language Models: A Surveycited by 14
  23. 2025Advancing Multi-agent Traffic Simulation via R1-Style Reinforcement Fine-Tuningcited by 14
  24. 2024AMP: Autoregressive Motion Prediction Revisited with Next Token Prediction for Autonomous Drivingcited by 13
  25. 2024Reliable Trajectory Prediction and Uncertainty Quantification with Conditioned Diffusion Modelscited by 13
  26. 2025UniMM: A Unified Mixture Model Framework for Multi-Agent Simulationcited by 13
  27. 2023Fine-Tuning Language Models Using Formal Methods Feedbackcited by 12
  28. 2025Generating Multimodal Driving Scenes via Next-Scene Predictioncited by 12
  29. 2025Decoupled Diffusion Sparks Adaptive Scene Generationcited by 12
  30. 2024SemanticFormer: Holistic and Semantic Traffic Scene Representation for Trajectory Prediction using Knowledge Graphscited by 11
  31. 2024Learning to Drive via Asymmetric Self-Playcited by 11
  32. 2023Categorical Traffic Transformer: Interpretable and Diverse Behavior Prediction with Tokenized Latentcited by 10
  33. 2025Unraveling Spatio-Temporal Foundation Models via the Pipeline Lens: A Comprehensive Reviewcited by 10
  34. 2026Bench2Drive-VL: Benchmarks for Closed-Loop Autonomous Driving with Vision-Language Modelscited by 10
  35. 2024Using Large Language Models to Automate and Expedite Reinforcement Learning with Reward Machinecited by 9
  36. 2024Words in Motion: Extracting Interpretable Control Vectors for Motion Transformerscited by 9
  37. 2024Curb Your Attention: Causal Attention Gating for Robust Trajectory Prediction in Autonomous Drivingcited by 9
  38. 2024World Models: The Safety Perspectivecited by 9

50 most cited of 156 All 156 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2021Scene Transformer: A unified architecture for predicting multiple agent trajectories
  2. 2024DeMo: Decoupling Motion Forecasting into Directional Intentions and Dynamic States
  3. 2026AutoTraces: Autoregressive Trajectory Forecasting via Multimodal Large Language Models
  4. 2022JFP: Joint Future Prediction with Interactive Multi-Agent Modeling for Autonomous Driving
  5. 2024Traj-LLM: A New Exploration for Empowering Trajectory Prediction with Pre-trained Large Language Models
  6. 2023MotionDiffuser: Controllable Multi-Agent Motion Prediction using Diffusion
  7. 2022Narrowing the Coordinate-frame Gap in Behavior Prediction Models: Distillation for Efficient and Accurate Scene-centric Motion Forecasting
  8. 2024Motion-Agent: A Conversational Framework for Human Motion Generation with LLMs