# GeoFitBench-LLM: A Benchmark for Assessing Large Language Models' Capabilities in Geospatial Data Requirements Elicitation for Geo-Analytic Tasks

arXiv: https://arxiv.org/abs/2610.12669 · HTML: https://mlsanity.com/paper/2610.12669/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 3 of its 12 references that are in the map.
- First posted on arXiv 8 Oct 2026.

## Builds on (most cited first, 3 of 3)

- [An Autonomous GIS Agent Framework for Geospatial Data Retrieval](https://mlsanity.com/paper/2407.21024/) (2024)
- [GeoBenchX: Benchmarking LLMs in Agent Solving Multistep Geospatial Tasks](https://mlsanity.com/paper/2503.18129/) (2025)
- [GeoAnalystBench: A GeoAI benchmark for assessing large language models for spatial analysis workflow and code generation](https://mlsanity.com/paper/2509.05881/) (2025)

## Abstract (arXiv)

Large language models are becoming central to GeoAI systems, yet their evaluation rarely considers the elicitation of geospatial data requirements. This omission is critical as retrieval oriented geo-analytic tasks cannot be supported reliably without first identifying the necessary datasets and constraints. GeoFitBench-LLM is introduced as a benchmark for assessing whether LLMs can infer the datasets, the supporting resources and the constraints required by geo-analytic requests in natural language. The benchmark is structured around two reasoning dimensions, Spatio-Temporal Relational and Semantic Grounding, reflecting long-standing challenges in geospatial information retrieval and related natural language tasks. Ground truth requirements are encoded in a fixed, provider-independent schema for reproducible evaluation of resource identification and attribute-level specification. Experiments on seven models under minimal prompting indicate that identifying the necessary resources is often achievable, with GPT-5.4 Thinking achieving the best result by recovering the complete set of resources in 66.67% of benchmark requests. Results further indicate that accurate fitness-for-use specification remains more difficult as inferential demands increase.