# Frozen LLMs as Map-Aware Spatio-Temporal Reasoners for Vehicle Trajectory Prediction

arXiv: https://arxiv.org/abs/2604.21479 · HTML: https://mlsanity.com/paper/2604.21479/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Placed in Chain-of-thought reasoning for driving, one of 17 directions in LLM reasoning.
- Builds on 12 of its 22 references that are in the map.
- First posted on arXiv 23 Apr 2026.

## Placed in

- LLM reasoning / [Chain-of-thought reasoning for driving](https://mlsanity.com/lr/chain-of-thought-reasoning-for-driving/)

## Builds on (most cited first, 12 of 12)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [nuScenes: A multimodal dataset for autonomous driving](https://mlsanity.com/paper/1903.11027/) (2019)
- [A Survey on Evaluation of Large Language Models](https://mlsanity.com/paper/2307.03109/) (2023)
- [Time-LLM: Time Series Forecasting by Reprogramming Large Language Models](https://mlsanity.com/paper/2310.01728/) (2023)
- [Spatial-Temporal Large Language Model for Traffic Prediction](https://mlsanity.com/paper/2401.10134/) (2024)
- [Logical Reasoning in Large Language Models: A Survey](https://mlsanity.com/paper/2502.09100/) (2025)
- [Advancing Reasoning in Large Language Models: Promising Methods and Approaches](https://mlsanity.com/paper/2502.03671/) (2025)
- [LSceneLLM: Enhancing Large 3D Scene Understanding Using Adaptive Visual Preferences](https://mlsanity.com/paper/2412.01292/) (2024)
- [Emergent Abilities in Large Language Models: A Survey](https://mlsanity.com/paper/2503.05788/) (2025)
- [Empowering Autonomous Driving with Large Language Models: A Safety Perspective](https://mlsanity.com/paper/2312.00812/) (2023)
- [Machine Learning for Autonomous Vehicle's Trajectory Prediction: A comprehensive survey, Challenges, and Future Research Directions](https://mlsanity.com/paper/2307.07527/) (2023)
- [Trajectory Prediction for Autonomous Driving: Progress, Limitations, and Future Directions](https://mlsanity.com/paper/2503.03262/) (2025)

## Abstract (arXiv)

Large language models (LLMs) have recently demonstrated strong reasoning capabilities and attracted increasing research attention in the field of autonomous driving (AD). However, safe application of LLMs on AD perception and prediction still requires a thorough understanding of both the dynamic traffic agents and the static road infrastructure. To this end, this study introduces a framework to evaluate the capability of LLMs in understanding the behaviors of dynamic traffic agents and the topology of road networks. The framework leverages frozen LLMs as the reasoning engine, employing a traffic encoder to extract spatial-level scene features from observed trajectories of agents, while a lightweight Convolutional Neural Network (CNN) encodes the local high-definition (HD) maps. To assess the intrinsic reasoning ability of LLMs, the extracted scene features are then transformed into LLM-compatible tokens via a reprogramming adapter. By residing the prediction burden with the LLMs, a simpler linear decoder is applied to output future trajectories. The framework enables a quantitative analysis of the influence of multi-modal information, especially the impact of map semantics on trajectory prediction accuracy, and allows seamless integration of frozen LLMs with minimal adaptation, thereby demonstrating strong generalizability across diverse LLM architectures and providing a unified platform for model evaluation.