# GeoBenchX: Benchmarking LLMs in Agent Solving Multistep Geospatial Tasks

arXiv: https://arxiv.org/abs/2503.18129 · HTML: https://mlsanity.com/paper/2503.18129/

## Key facts

- Cited by 14 papers in the map (data through 14 Oct 2026).
- Placed in Multimodal and embodied agents, one of 16 directions in LLM agents and tool use.
- Builds on 10 of its 29 references that are in the map.
- First posted on arXiv 23 Mar 2025.

## Placed in

- LLM agents and tool use / [Multimodal and embodied agents](https://mlsanity.com/agents/multimodal-and-embodied-agents/)

## Builds on (most cited first, 10 of 10)

- [Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena](https://mlsanity.com/paper/2306.05685/) (2023)
- [Can Large Language Models Be an Alternative to Human Evaluations?](https://mlsanity.com/paper/2305.01937/) (2023)
- [GPT4GEO: How a Language Model Sees the World's Geography](https://mlsanity.com/paper/2306.00020/) (2023)
- [Autonomous GIS: the next-generation AI-powered GIS](https://mlsanity.com/paper/2305.06453/) (2023)
- [Are Large Language Models Geospatially Knowledgeable?](https://mlsanity.com/paper/2310.13002/) (2023)
- [GIS Copilot: Towards an Autonomous GIS Agent for Spatial Analysis](https://mlsanity.com/paper/2411.03205/) (2024)
- [FixEval: Execution-based Evaluation of Program Fixes for Programming Problems](https://mlsanity.com/paper/2206.07796/) (2022)
- [Evaluating Large Language Models on Spatial Tasks: A Multi-Task Benchmarking Study](https://mlsanity.com/paper/2408.14438/) (2024)
- [ShapefileGPT: A Multi-Agent Large Language Model Framework for Automated Shapefile Processing](https://mlsanity.com/paper/2410.12376/) (2024)
- [Evaluation of Code LLMs on Geospatial Code Generation](https://mlsanity.com/paper/2410.04617/) (2024)

## Built on by (most cited first, 14 of 14)

- [Agentic AI in Remote Sensing: Foundations, Taxonomy, and Emerging Systems](https://mlsanity.com/paper/2601.01891/) (2026)
- [GeoAgentBench: A Dynamic Execution Benchmark for Tool-Augmented Agents in Spatial Analysis](https://mlsanity.com/paper/2604.13888/) (2026)
- [TerraBench: Can Agents Reason Over Heterogeneous Earth-System Data?](https://mlsanity.com/paper/2606.13148/) (2026)
- [GPSBench: Do Large Language Models Understand GPS Coordinates?](https://mlsanity.com/paper/2602.16105/) (2026)
- [GeoNatureAgent Benchmark: Benchmarking LLM Agents for Environmental Geospatial Analysis Across Frontier and Open-Weight Foundation Models](https://mlsanity.com/paper/2606.12821/) (2026)
- [AgentJudgeBench: A Multi-Difficulty Benchmark for Evaluating LLM Judges on Agentic Tool-Calling](https://mlsanity.com/paper/2608.26623/) (2026)
- [GISAgentBench: A Practitioner-Sourced Benchmark for Evaluating LLM Agents on GIS Tasks](https://mlsanity.com/paper/2608.01645/) (2026)
- [Planetary Prediction Engine: Autonomous Geospatial Prediction via Intelligent Data Selection and Foundation Model Embeddings](https://mlsanity.com/paper/2608.26088/) (2026)
- [ANASSA: An Agentic AI Orchestration Framework for Spatial Intelligence](https://mlsanity.com/paper/2609.14824/) (2026)
- [SimCRAFT: Distilling Remote Sensing Agents via Synthetic Trajectories and Contextual Retrieval-Augmented Fine-Tuning](https://mlsanity.com/paper/2608.30277/) (2026)
- [Earth-Agent-Pro: Towards Real-World Full-Chain Earth Observation with Agents](https://mlsanity.com/paper/2609.12533/) (2026)
- [GeoFitBench-LLM: A Benchmark for Assessing Large Language Models' Capabilities in Geospatial Data Requirements Elicitation for Geo-Analytic Tasks](https://mlsanity.com/paper/2610.12669/) (2026)
- [Google Earth AI Geospatial Reasoning Agent: Capabilities and a Systematic Evaluation](https://mlsanity.com/paper/2610.15750/) (2026)
- [Trust Before Autonomy: Rethinking Agentic Remote Sensing for Crisis Response](https://mlsanity.com/paper/2610.18789/) (2026)

## Abstract (arXiv)

This paper establishes a benchmark for evaluating tool-calling capabilities of large language models (LLMs) on multi-step geospatial tasks relevant to commercial GIS practitioners. We assess eight commercial LLMs (Claude Sonnet 3.5 and 4, Claude Haiku 3.5, Gemini 2.0 Flash, Gemini 2.5 Pro Preview, GPT-4o, GPT-4.1 and o4-mini) using a simple tool-calling agent equipped with 23 geospatial functions. Our benchmark comprises tasks in four categories of increasing complexity, with both solvable and intentionally unsolvable tasks to test rejection accuracy. We develop a LLM-as-Judge evaluation framework to compare agent solutions against reference solutions. Results show o4-mini and Claude 3.5 Sonnet achieve the best overall performance, OpenAI's GPT-4.1, GPT-4o and Google's Gemini 2.5 Pro Preview do not fall far behind, but the last two are more efficient in identifying unsolvable tasks. Claude Sonnet 4, due its preference to provide any solution rather than reject a task, proved to be less accurate. We observe significant differences in token usage, with Anthropic models consuming more tokens than competitors. Common errors include misunderstanding geometrical relationships, relying on outdated knowledge, and inefficient data manipulation. The resulting benchmark set, evaluation framework, and data generation pipeline are released as open-source resources (available at https://github.com/Solirinai/GeoBenchX), providing one more standardized method for the ongoing evaluation of LLMs for GeoAI.