# How Can Large Language Models Understand Spatial-Temporal Data?

arXiv: https://arxiv.org/abs/2401.14192 · HTML: https://mlsanity.com/paper/2401.14192/

## Key facts

- Cited by 33 papers in the map (data through 14 Oct 2026).
- Builds on 20 of its 33 references that are in the map.
- First posted on arXiv 25 Jan 2024.

## Builds on (most cited first, 20 of 20)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [LLaMA: Open and Efficient Foundation Language Models](https://mlsanity.com/paper/2302.13971/) (2023)
- [Graph Attention Networks](https://mlsanity.com/paper/1710.10903/) (2017)
- [Parameter-Efficient Transfer Learning for NLP](https://mlsanity.com/paper/1902.00751/) (2019)
- [InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuning](https://mlsanity.com/paper/2305.06500/) (2023)
- [Spatio-Temporal Graph Convolutional Networks: A Deep Learning Framework for Traffic Forecasting](https://mlsanity.com/paper/1709.04875/) (2017)
- [Graph WaveNet for Deep Spatial-Temporal Graph Modeling](https://mlsanity.com/paper/1906.00121/) (2019)
- [Time-LLM: Time Series Forecasting by Reprogramming Large Language Models](https://mlsanity.com/paper/2310.01728/) (2023)
- [One Fits All:Power General Time Series Analysis by Pretrained LM](https://mlsanity.com/paper/2302.11939/) (2023)
- [Adaptive Graph Convolutional Recurrent Network for Traffic Forecasting](https://mlsanity.com/paper/2007.02842/) (2020)
- [Large Language Models Are Zero-Shot Time Series Forecasters](https://mlsanity.com/paper/2310.07820/) (2023)
- [Spatial-Temporal Fusion Graph Neural Networks for Traffic Flow Forecasting](https://mlsanity.com/paper/2012.09641/) (2020)
- [Spatial-Temporal Graph ODE Networks for Traffic Flow Forecasting](https://mlsanity.com/paper/2106.12931/) (2021)
- [Pretrained Transformers as Universal Computation Engines](https://mlsanity.com/paper/2103.05247/) (2021)
- [PDFormer: Propagation Delay-Aware Dynamic Long-Range Transformer for Traffic Flow Prediction](https://mlsanity.com/paper/2301.07945/) (2023)
- [Recommender AI Agent: Integrating Large Language Models for Interactive Recommendations](https://mlsanity.com/paper/2308.16505/) (2023)
- [Exploring Progress in Multivariate Time Series Forecasting: Comprehensive Benchmarking and Heterogeneity Analysis](https://mlsanity.com/paper/2310.06119/) (2023)
- [Is ChatGPT Fair for Recommendation? Evaluating Fairness in Large Language Model Recommendation](https://mlsanity.com/paper/2305.07609/) (2023)
- [GIMLET: A Unified Graph-Text Model for Instruction-Based Molecule Zero-Shot Learning](https://mlsanity.com/paper/2306.13089/) (2023)
- [Towards Language Models That Can See: Computer Vision Through the LENS of Natural Language](https://mlsanity.com/paper/2306.16410/) (2023)

## Built on by (most cited first, 30 of 33)

- [UniST: A Prompt-Empowered Universal Model for Urban Spatio-Temporal Prediction](https://mlsanity.com/paper/2402.11838/) (2024)
- [Exploring the Roles of Large Language Models in Reshaping Transportation Systems: A Survey, Framework, and Roadmap](https://mlsanity.com/paper/2503.21411/) (2025)
- [Foundation Models for Spatio-Temporal Data Science: A Tutorial and Survey](https://mlsanity.com/paper/2503.13502/) (2025)
- [Multi-modal Time Series Analysis: A Tutorial and Survey](https://mlsanity.com/paper/2503.13709/) (2025)
- [Towards Urban General Intelligence: A Review and Outlook of Urban Foundation Models](https://mlsanity.com/paper/2402.01749/) (2024)
- [Empowering Time Series Analysis with Foundation Models: A Comprehensive Survey](https://mlsanity.com/paper/2405.02358/) (2024)
- [Integrating LLMs with ITS: Recent Advances, Potentials, Challenges, and Future Directions](https://mlsanity.com/paper/2501.04437/) (2025)
- [How Can Time Series Analysis Benefit From Multiple Modalities? A Survey and Outlook](https://mlsanity.com/paper/2503.11835/) (2025)
- [BCAT: A Block Causal Transformer for PDE Foundation Models for Fluid Dynamics](https://mlsanity.com/paper/2501.18972/) (2025)
- [Large Language Models for Mobility Analysis in Transportation Systems: A Survey on Forecasting Tasks](https://mlsanity.com/paper/2405.02357/) (2024)
- [Online Reasoning Video Segmentation with Just-in-Time Digital Twins](https://mlsanity.com/paper/2503.21056/) (2025)
- [STD-PLM: Understanding Both Spatial and Temporal Properties of Spatial-Temporal Data with PLM](https://mlsanity.com/paper/2407.09096/) (2024)
- [Unraveling Spatio-Temporal Foundation Models via the Pipeline Lens: A Comprehensive Review](https://mlsanity.com/paper/2506.01364/) (2025)
- [LLM-driven Knowledge Distillation for Dynamic Text-Attributed Graphs](https://mlsanity.com/paper/2502.10914/) (2025)
- [CoMaPOI: A Collaborative Multi-Agent Framework for Next POI Prediction Bridging the Gap Between Trajectory and Language](https://mlsanity.com/paper/2505.23837/) (2025)
- [UniFlow: A Foundation Model for Unified Urban Spatio-Temporal Flow Prediction](https://mlsanity.com/paper/2411.12972/) (2024)
- [Text Reinforcement for Multimodal Time Series Forecasting](https://mlsanity.com/paper/2509.00687/) (2025)
- [STReasoner: Empowering LLMs for Spatio-Temporal Reasoning in Time Series via Spatial-Aware Reinforcement Learning](https://mlsanity.com/paper/2601.03248/) (2026)
- [T-GRAG: A Dynamic GraphRAG Framework for Resolving Temporal Conflicts and Redundancy in Knowledge Retrieval](https://mlsanity.com/paper/2508.01680/) (2025)
- [Large Foundation Models for Trajectory Prediction in Autonomous Driving: A Comprehensive Survey](https://mlsanity.com/paper/2509.10570/) (2025)
- [RePST: Language Model Empowered Spatio-Temporal Forecasting via Semantic-Oriented Reprogramming](https://mlsanity.com/paper/2408.14505/) (2024)
- [Urban Computing in the Era of Large Language Models](https://mlsanity.com/paper/2504.02009/) (2025)
- [UrbanMind: Urban Dynamics Prediction with Multifaceted Spatial-Temporal Large Language Models](https://mlsanity.com/paper/2505.11654/) (2025)
- [Using LLMs for Analyzing AIS Data](https://mlsanity.com/paper/2504.07557/) (2025)
- [Vision-LLMs for Spatiotemporal Traffic Forecasting](https://mlsanity.com/paper/2510.11282/) (2025)
- [Mixer is more than just a model](https://mlsanity.com/paper/2402.18007/) (2024)
- [IMAIA: Interactive Maps AI Assistant for Travel Planning and Geo-Spatial Intelligence](https://mlsanity.com/paper/2507.06993/) (2025)
- [Reprogramming Vision Foundation Models for Spatio-Temporal Forecasting](https://mlsanity.com/paper/2507.11558/) (2025)
- [Comprehending Spatio-temporal Data via Cinematic Storytelling using Large Language Models](https://mlsanity.com/paper/2510.17301/) (2025)
- [LLMComp: A Language Modeling Paradigm for Error-Bounded Scientific Data Compression (Technical Report)](https://mlsanity.com/paper/2510.23632/) (2025)

## Abstract (arXiv)

While Large Language Models (LLMs) dominate tasks like natural language processing and computer vision, harnessing their power for spatial-temporal forecasting remains challenging. The disparity between sequential text and complex spatial-temporal data hinders this application. To address this issue, this paper introduces STG-LLM, an innovative approach empowering LLMs for spatial-temporal forecasting. We tackle the data mismatch by proposing: 1) STG-Tokenizer: This spatial-temporal graph tokenizer transforms intricate graph data into concise tokens capturing both spatial and temporal relationships; 2) STG-Adapter: This minimalistic adapter, consisting of linear encoding and decoding layers, bridges the gap between tokenized data and LLM comprehension. By fine-tuning only a small set of parameters, it can effectively grasp the semantics of tokens generated by STG-Tokenizer, while preserving the original natural language understanding capabilities of LLMs. Extensive experiments on diverse spatial-temporal benchmark datasets show that STG-LLM successfully unlocks LLM potential for spatial-temporal forecasting. Remarkably, our approach achieves competitive performance on par with dedicated SOTA methods.