mlsanityData to 14 Oct 2026

Trajectory Prediction Meets Large Language Models: A Survey

arXiv 2506.03408 ↗First posted 3 Jun 2025cs.CL cs.CVcited by 14

Not on the map. Why?

Abstract (arXiv)

Recent advances in large language models (LLMs) have sparked growing interest in integrating language-driven techniques into trajectory prediction. By leveraging their semantic and reasoning capabilities, LLMs are reshaping how autonomous systems perceive, model, and predict trajectories. This survey provides a comprehensive overview of this emerging field, categorizing recent work into five directions: (1) Trajectory prediction via language modeling paradigms, (2) Direct trajectory prediction with pretrained language models, (3) Language-guided scene understanding for trajectory prediction, (4) Language-driven data generation for trajectory prediction, (5) Language-based reasoning and interpretability for trajectory prediction. For each, we analyze representative methods, highlight core design choices, and identify open challenges. This survey bridges natural language processing and trajectory prediction, offering a unified perspective on how language can enrich trajectory prediction.

Key facts. Cited by 14 papers in the map (data through 14 Oct 2026). Builds on 42 of its 70 references that are in the map. First posted on arXiv 3 Jun 2025.

Builds on

42 of 70 references.

References · 42

  1. 2017Attention Is All You Needcited by 60,361
  2. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  3. 2023Mamba: Linear-Time Sequence Modeling with Selective State Spacescited by 4,718
  4. 2019nuScenes: A multimodal dataset for autonomous drivingcited by 4,523
  5. 2022Large Language Models are Zero-Shot Reasonerscited by 4,374
  6. 2019DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lightercited by 4,307
  7. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,780
  8. 2019Argoverse: 3D Tracking and Forecasting with Rich Mapscited by 732
  9. 2023Argoverse 2: Next Generation Datasets for Self-Driving Perception and Forecastingcited by 707
  10. 2021NuPlan: A closed-loop ML-based planning benchmark for autonomous vehiclescited by 493
  11. 2023GPT-Driver: Learning to Drive with GPTcited by 347
  12. 2023A Survey on Multimodal Large Language Models for Autonomous Drivingcited by 290
30 more
  1. 2022Wayformer: Motion Forecasting via Simple & Efficient Attention Networkscited by 280
  2. 2018The highD Dataset: A Drone Dataset of Naturalistic Vehicle Trajectories on German Highways for Validation of Highly Automated Driving Systemscited by 277
  3. 2022Motion Transformer with Global Intention Localization and Local Movement Refinementcited by 270
  4. 2023Driving with LLMs: Fusing Object-Level Vector Modality for Explainable Autonomous Drivingcited by 232
  5. 2019INTERACTION Dataset: An INTERnational, Adversarial and Cooperative moTION Dataset in Interactive Driving Scenarios with Semantic Mapscited by 231
  6. 2018Convolutional Social Pooling for Vehicle Trajectory Predictioncited by 221
  7. 2021Multimodal Motion Prediction with Stacked Transformerscited by 208
  8. 2021Scene Transformer: A unified architecture for predicting multiple agent trajectoriescited by 165
  9. 2023MotionLM: Multi-Agent Motion Forecasting as Language Modelingcited by 156
  10. 2023MTR++: Multi-Agent Motion Prediction with Symmetric Scene Modeling and Guided Intention Queryingcited by 140
  11. 2023LLM4Drive: A Survey of Large Language Models for Autonomous Drivingcited by 136
  12. 2019Multi-Agent Tensor Fusion for Contextual Trajectory Predictioncited by 131
  13. 2023Vision Language Models in Autonomous Driving: A Survey and Outlookcited by 126
  14. 2023LargeST: A Benchmark Dataset for Large-Scale Traffic Forecastingcited by 118
  15. 2023Language-Guided Traffic Simulation via Scene-Level Diffusioncited by 111
  16. 2024Spatial-Temporal Large Language Model for Traffic Predictioncited by 83
  17. 2023Trajeglish: Traffic Modeling as Next-Token Predictioncited by 60
  18. 2024HPNet: Dynamic Trajectory Forecasting with Historical Prediction Attentioncited by 38
  19. 2024Can Language Beat Numerical Regression? Language-Based Multimodal Trajectory Predictioncited by 31
  20. 2023Can you text what is happening? Integrating pre-trained language encoders into trajectory prediction models for autonomous drivingcited by 30
  21. 2023BAT: Behavior-Aware Human-Like Trajectory Prediction for Autonomous Drivingcited by 28
  22. 2024Language-Driven Interactive Traffic Trajectory Generationcited by 19
  23. 2024Sports-Traj: A Unified Trajectory Generation Model for Multi-Agent Movement in Sportscited by 19
  24. 2024VisionTrap: Vision-Augmented Trajectory Prediction Guided by Textual Descriptionscited by 19
  25. 2024DeMo: Decoupling Motion Forecasting into Directional Intentions and Dynamic Statescited by 17
  26. 2023Trajectory Prediction for Robot Navigation using Flow-Guided Markov Neural Operatorcited by 6
  27. 2024Scaling Motion Forecasting Models with Ensemble Distillationcited by 6
  28. 2025Large Language Models for Single-Step and Multi-Step Flight Trajectory Predictioncited by 5
  29. 2023Interpretable Long Term Waypoint-Based Trajectory Prediction Modelcited by 1
  30. 2025Knowledge Integration Strategies in Autonomous Vehicle Prediction and Planning: A Comprehensive Surveycited by 1

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 5; 2026: 9.

14 papers.

Sort:

By area:

  1. 2026From Chatbot to Digital Colleague: The Paradigm Shift Toward Persistent Autonomous AIcited by 6
  2. 2025A Benchmark for Procedural Memory Retrieval in Language Agentscited by 4
  3. 2026Post-Training in End-to-End Autonomous Drivingcited by 2
  4. 2025SVBRD-LLM: Self-Verifying Behavioral Rule Discovery for Autonomous Vehicle Identificationcited by 1
  5. 2026VISTA: Knowledge-Driven Vessel Trajectory Imputation with Repair Provenancecited by 1
  6. 2026Evaluating randomized smoothing as a defense against adversarial attacks in trajectory predictioncited by 1
  7. 2026Learning Probabilistic Responsibility Allocations for Multi-Agent Interactionscited by 1
  8. 2025LinguaFluid: Language Guided Fluid Control via Semantic Rewards in Reinforcement Learningcited by 0
  9. 2025Research on Short-Video Platform User Decision-Making via Multimodal Temporal Modeling and Reinforcement Learningcited by 0
  10. 2025Are VLMs Ready for Lane Topology Awareness in Autonomous Driving?cited by 0
  11. 2026Context-Enriched Natural Language Descriptions of Vessel Trajectoriescited by 0
  12. 2026General-purpose LLMs as Models of Human Driver Behavior: The Case of Simplified Mergingcited by 0
2 more
  1. 2026M3-Former: Multimodal Transformer with Mixture-of-Experts for Long-Term Vessel Trajectory Predictioncited by 0
  2. 2026Industrial Kinematic Trajectory Model (IKTM): Coordinate-Free Autoregressive Generatorcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026AutoTraces: Autoregressive Trajectory Forecasting via Multimodal Large Language Models
  2. 2025Large Foundation Models for Trajectory Prediction in Autonomous Driving: A Comprehensive Survey
  3. 2024Traj-LLM: A New Exploration for Empowering Trajectory Prediction with Pre-trained Large Language Models
  4. 2026BitTP: The Lightweight Trajectory Prediction Model with BitLLM for Edge-Devices
  5. 2022Reshaping Robot Trajectories Using Natural Language Commands: A Study of Multi-Modal Data Alignment Using Transformers
  6. 2025LLaViDA: A Large Language Vision Driving Assistant for Explicit Reasoning and Enhanced Trajectory Planning
  7. 2023GG-LLM: Geometrically Grounding Large Language Models for Zero-shot Human Activity Forecasting in Human-Aware Task Planning
  8. 2026WorldMAP: Bootstrapping Vision-Language Navigation Trajectory Prediction with Generative World Models