# GeoBenchLLM: A Comprehensive Benchmark for Evaluating LLMs on Geo-Related Tasks

arXiv: https://arxiv.org/abs/2608.07411 · HTML: https://mlsanity.com/paper/2608.07411/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 11 of its 23 references that are in the map.
- First posted on arXiv 7 Aug 2026.

## Builds on (most cited first, 11 of 11)

- [MS MARCO: A Human Generated MAchine Reading COmprehension Dataset](https://mlsanity.com/paper/1611.09268/) (2016)
- [Towards AI-Complete Question Answering: A Set of Prerequisite Toy Tasks](https://mlsanity.com/paper/1502.05698/) (2015)
- [Improving Text-to-SQL Evaluation Methodology](https://mlsanity.com/paper/1806.09029/) (2018)
- [StepGame: A New Benchmark for Robust Multi-Hop Spatial Reasoning in Texts](https://mlsanity.com/paper/2204.08292/) (2022)
- [GETNext: Trajectory Flow Map Enhanced Transformer for Next POI Recommendation](https://mlsanity.com/paper/2303.04741/) (2023)
- [Can Large Language Models be Good Path Planners? A Benchmark and Investigation on Spatial-temporal Reasoning](https://mlsanity.com/paper/2310.03249/) (2023)
- [Transfer Learning with Synthetic Corpora for Spatial Role Labeling and Reasoning](https://mlsanity.com/paper/2210.16952/) (2022)
- [Geographic Question Answering: Challenges, Uniqueness, Classification, and Future Directions](https://mlsanity.com/paper/2105.09392/) (2021)
- [GeoSQA: A Benchmark for Scenario-based Question Answering in the Geography Domain at High School Level](https://mlsanity.com/paper/1908.07855/) (2019)
- [MapQA: Open-domain Geospatial Question Answering on Map Data](https://mlsanity.com/paper/2503.07871/) (2025)
- [When Retriever-Reader Meets Scenario-Based Multiple-Choice Questions](https://mlsanity.com/paper/2108.13875/) (2021)

## Abstract (arXiv)

In the context of geodata, existing Large Language Models have often been studied in a homogeneous setting, which has considerably limited insights into their generalization capabilities. In this paper, we present benchName, a comprehensive benchmark for probing LLMs on geo-related tasks. We leverage a careful selection of twelve publicly available datasets from diverse geo-related tasks and domains, and evaluate a set of LLMs on geo-spatial and temporal understanding using our benchmark. Our results show that reasoning and size have a strong impact on overall performance. GeoBenchLLM is publicly available at https://github.com/Rfr2003/GeoBenchLLM.