# Correctness Comparison of ChatGPT-4, Gemini, Claude-3, and Copilot for Spatial Tasks

arXiv: https://arxiv.org/abs/2401.02404 · HTML: https://mlsanity.com/paper/2401.02404/

## Key facts

- Cited by 10 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 4 Jan 2024.

## Built on by (most cited first, 10 of 10)

- [Evaluating Large Language Models on Spatial Tasks: A Multi-Task Benchmarking Study](https://mlsanity.com/paper/2408.14438/) (2024)
- [GUIDE: Graphical User Interface Data for Execution](https://mlsanity.com/paper/2404.16048/) (2024)
- [MapIQ: Evaluating Multimodal Large Language Models for Map Question Answering](https://mlsanity.com/paper/2507.11625/) (2025)
- [Geospatial Mechanistic Interpretability of Large Language Models](https://mlsanity.com/paper/2505.03368/) (2025)
- [Assessing the Geolocation Capabilities, Limitations and Societal Risks of Generative Vision-Language Models](https://mlsanity.com/paper/2508.19967/) (2025)
- [3DFroMLLM: 3D Prototype Generation only from Pretrained Multimodal LLMs](https://mlsanity.com/paper/2508.08821/) (2025)
- [A Dual-Helix Governance Approach Towards Reliable Agentic Artificial Intelligence for WebGIS Development](https://mlsanity.com/paper/2603.04390/) (2026)
- [Vision language models have difficulty recognizing virtual objects](https://mlsanity.com/paper/2505.10453/) (2025)
- [Large Multi-modal Model Cartographic Map Comprehension for Textual Locality Georeferencing](https://mlsanity.com/paper/2507.08575/) (2025)
- [A cybersecurity AI agent selection and decision support framework](https://mlsanity.com/paper/2510.01751/) (2025)

## Abstract (arXiv)

Generative AI including large language models (LLMs) has recently gained significant interest in the geo-science community through its versatile task-solving capabilities including programming, arithmetic reasoning, generation of sample data, time-series forecasting, toponym recognition, or image classification. Most existing performance assessments of LLMs for spatial tasks have primarily focused on ChatGPT, whereas other chatbots received less attention. To narrow this research gap, this study conducts a zero-shot correctness evaluation for a set of 76 spatial tasks across seven task categories assigned to four prominent chatbots, i.e., ChatGPT-4, Gemini, Claude-3, and Copilot. The chatbots generally performed well on tasks related to spatial literacy, GIS theory, and interpretation of programming code and functions, but revealed weaknesses in mapping, code writing, and spatial reasoning. Furthermore, there was a significant difference in correctness of results between the four chatbots. Responses from repeated tasks assigned to each chatbot showed a high level of consistency in responses with matching rates of over 80% for most task categories in the four chatbots.