# Are Large Language Models Geospatially Knowledgeable?

arXiv: https://arxiv.org/abs/2310.13002 · HTML: https://mlsanity.com/paper/2310.13002/

## Key facts

- Cited by 30 papers in the map (data through 14 Oct 2026).
- Builds on 15 of its 30 references that are in the map.
- First posted on arXiv 9 Oct 2023.

## Builds on (most cited first, 15 of 15)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Scaling Laws for Neural Language Models](https://mlsanity.com/paper/2001.08361/) (2020)
- [Efficient Estimation of Word Representations in Vector Space](https://mlsanity.com/paper/1301.3781/) (2013)
- [Prefix-Tuning: Optimizing Continuous Prompts for Generation](https://mlsanity.com/paper/2101.00190/) (2021)
- [HuggingFace's Transformers: State-of-the-art Natural Language Processing](https://mlsanity.com/paper/1910.03771/) (2019)
- [Language Models as Knowledge Bases?](https://mlsanity.com/paper/1909.01066/) (2019)
- [Stanza: A Python Natural Language Processing Toolkit for Many Human Languages](https://mlsanity.com/paper/2003.07082/) (2020)
- [P-Tuning v2: Prompt Tuning Can Be Comparable to Fine-tuning Universally Across Scales and Tasks](https://mlsanity.com/paper/2110.07602/) (2021)
- [CCNet: Extracting High Quality Monolingual Datasets from Web Crawl Data](https://mlsanity.com/paper/1911.00359/) (2019)
- [Dissecting Contextual Word Embeddings: Architecture and Representation](https://mlsanity.com/paper/1808.08949/) (2018)
- [Analyzing Commonsense Emergence in Few-shot Knowledge Models](https://mlsanity.com/paper/2101.00297/) (2021)
- [Relational World Knowledge Representation in Contextual Language Models: A Review](https://mlsanity.com/paper/2104.05837/) (2021)
- [Do Language Models Know the Way to Rome?](https://mlsanity.com/paper/2109.07971/) (2021)

## Built on by (most cited first, 30 of 30)

- [CityGPT: Empowering Urban Spatial Cognition of Large Language Models](https://mlsanity.com/paper/2406.13948/) (2024)
- [CityBench: Evaluating the Capabilities of Large Language Models for Urban Tasks](https://mlsanity.com/paper/2406.13945/) (2024)
- [Towards Urban General Intelligence: A Review and Outlook of Urban Foundation Models](https://mlsanity.com/paper/2402.01749/) (2024)
- [UrbanKGent: A Unified Large Language Model Agent Framework for Urban Knowledge Graph Construction](https://mlsanity.com/paper/2402.06861/) (2024)
- [AI Sees Your Location, But With A Bias Toward The Wealthy World](https://mlsanity.com/paper/2502.11163/) (2025)
- [GeoBenchX: Benchmarking LLMs in Agent Solving Multistep Geospatial Tasks](https://mlsanity.com/paper/2503.18129/) (2025)
- [GeoLocator: a location-integrated large multimodal model for inferring geo-privacy](https://mlsanity.com/paper/2311.13018/) (2023)
- [Urban Mobility Assessment Using LLMs](https://mlsanity.com/paper/2409.00063/) (2024)
- [AeroDuo: Aerial Duo for UAV-based Vision and Language Navigation](https://mlsanity.com/paper/2508.15232/) (2025)
- [A Survey of Large Language Model-Powered Spatial Intelligence Across Scales: Advances in Embodied Agents, Smart Cities, and Earth Science](https://mlsanity.com/paper/2504.09848/) (2025)
- [Richer Output for Richer Countries: Uncovering Geographical Disparities in Generated Stories and Travel Recommendations](https://mlsanity.com/paper/2411.07320/) (2024)
- [Casper: Prompt Sanitization for Protecting User Privacy in Web-Based Large Language Models](https://mlsanity.com/paper/2408.07004/) (2024)
- [Evaluation of Code LLMs on Geospatial Code Generation](https://mlsanity.com/paper/2410.04617/) (2024)
- [Quantifying Geospatial in the Common Crawl Corpus](https://mlsanity.com/paper/2406.04952/) (2024)
- [Urban Computing in the Era of Large Language Models](https://mlsanity.com/paper/2504.02009/) (2025)
- [Geospatial Mechanistic Interpretability of Large Language Models](https://mlsanity.com/paper/2505.03368/) (2025)
- [GeoGrid-Bench: Can Foundation Models Understand Multimodal Gridded Geo-Spatial Data?](https://mlsanity.com/paper/2505.10714/) (2025)
- [The World As Large Language Models See It: Exploring the reliability of LLMs in representing geographical features](https://mlsanity.com/paper/2506.00203/) (2025)
- [GeoResponder: Towards Building Geospatial LLMs for Time-Critical Disaster Response](https://mlsanity.com/paper/2509.19354/) (2025)
- [CartoMapQA: A Fundamental Benchmark Dataset Evaluating Vision-Language Models on Cartographic Map Understanding](https://mlsanity.com/paper/2512.03558/) (2025)
- [Georeferencing complex relative locality descriptions with large language models](https://mlsanity.com/paper/2512.14228/) (2025)
- [Benchmarks for Vision-Language Models in Urban Perception Should Be Reliability-Aware and Negotiated](https://mlsanity.com/paper/2606.00871/) (2026)
- [MALAMUTE: A Multilingual, Highly-granular, Template-free, Education-based Probing Dataset](https://mlsanity.com/paper/2412.10105/) (2024)
- [TurnBack: A Geospatial Route Cognition Benchmark for Large Language Models through Reverse Route](https://mlsanity.com/paper/2509.18173/) (2025)
- [Coordinates from Context: Using LLMs to Ground Complex Location References](https://mlsanity.com/paper/2510.08741/) (2025)
- [GTPred: Benchmarking MLLMs for Interpretable Geo-localization and Time-of-capture Prediction](https://mlsanity.com/paper/2601.13207/) (2026)
- [Much of Geospatial Web Search Is Beyond Traditional GIS](https://mlsanity.com/paper/2605.11336/) (2026)
- [Assessing the Geographic Diversity of AI's Platial Representations in Image Generation](https://mlsanity.com/paper/2606.05188/) (2026)
- [Graph-Enhanced Large Language Models for Spatial Search](https://mlsanity.com/paper/2606.22909/) (2026)
- [Is Your Neighborhood Safe? Place-based Stigma in Large Language Models' Urban Safety Judgments](https://mlsanity.com/paper/2608.26188/) (2026)

## Abstract (arXiv)

Despite the impressive performance of Large Language Models (LLM) for various natural language processing tasks, little is known about their comprehension of geographic data and related ability to facilitate informed geospatial decision-making. This paper investigates the extent of geospatial knowledge, awareness, and reasoning abilities encoded within such pretrained LLMs. With a focus on autoregressive language models, we devise experimental approaches related to (i) probing LLMs for geo-coordinates to assess geospatial knowledge, (ii) using geospatial and non-geospatial prepositions to gauge their geospatial awareness, and (iii) utilizing a multidimensional scaling (MDS) experiment to assess the models' geospatial reasoning capabilities and to determine locations of cities based on prompting. Our results confirm that it does not only take larger, but also more sophisticated LLMs to synthesize geospatial knowledge from textual information. As such, this research contributes to understanding the potential and limitations of LLMs in dealing with geospatial information.