# GeoLLM-Engine: A Realistic Environment for Building Geospatial Copilots

arXiv: https://arxiv.org/abs/2404.15500 · HTML: https://mlsanity.com/paper/2404.15500/

## Key facts

- Cited by 30 papers in the map (data through 14 Oct 2026).
- Builds on 30 of its 47 references that are in the map.
- First posted on arXiv 23 Apr 2024.

## Builds on (most cited first, 30 of 30)

- [Chain-of-Thought Prompting Elicits Reasoning in Large Language Models](https://mlsanity.com/paper/2201.11903/) (2022)
- [Self-Consistency Improves Chain of Thought Reasoning in Language Models](https://mlsanity.com/paper/2203.11171/) (2022)
- [Sparks of Artificial General Intelligence: Early experiments with GPT-4](https://mlsanity.com/paper/2303.12712/) (2023)
- [WebArena: A Realistic Web Environment for Building Autonomous Agents](https://mlsanity.com/paper/2307.13854/) (2023)
- [OpenAI Gym](https://mlsanity.com/paper/1606.01540/) (2016)
- [WebShop: Towards Scalable Real-World Web Interaction with Grounded Language Agents](https://mlsanity.com/paper/2207.01206/) (2022)
- [TOFU: A Task of Fictitious Unlearning for LLMs](https://mlsanity.com/paper/2401.06121/) (2024)
- [MM-REACT: Prompting ChatGPT for Multimodal Reasoning and Action](https://mlsanity.com/paper/2303.11381/) (2023)
- [Chameleon: Plug-and-Play Compositional Reasoning with Large Language Models](https://mlsanity.com/paper/2304.09842/) (2023)
- [Functional Map of the World](https://mlsanity.com/paper/1711.07846/) (2017)
- [LLMLingua: Compressing Prompts for Accelerated Inference of Large Language Models](https://mlsanity.com/paper/2310.05736/) (2023)
- [Reinforcement Learning on Web Interfaces Using Workflow-Guided Exploration](https://mlsanity.com/paper/1802.08802/) (2018)
- [ToolQA: A Dataset for LLM Question Answering with External Tools](https://mlsanity.com/paper/2306.13304/) (2023)
- [BigEarthNet: A Large-Scale Benchmark Archive For Remote Sensing Image Understanding](https://mlsanity.com/paper/1902.06148/) (2019)
- [RSVQA: Visual Question Answering for Remote Sensing Data](https://mlsanity.com/paper/2003.07333/) (2020)
- [Exploring Models and Data for Remote Sensing Image Caption Generation](https://mlsanity.com/paper/1712.07835/) (2017)
- [EarthGPT: A Universal Multi-modal Large Language Model for Multi-sensor Image Comprehension in Remote Sensing Domain](https://mlsanity.com/paper/2401.16822/) (2024)
- [An LLM Compiler for Parallel Function Calling](https://mlsanity.com/paper/2312.04511/) (2023)
- [SatlasPretrain: A Large-Scale Dataset for Remote Sensing Image Understanding](https://mlsanity.com/paper/2211.15660/) (2022)
- [xView: Objects in Context in Overhead Imagery](https://mlsanity.com/paper/1802.07856/) (2018)
- [FAIR1M: A Benchmark Dataset for Fine-grained Object Recognition in High-Resolution Remote Sensing Imagery](https://mlsanity.com/paper/2103.05569/) (2021)
- [LHRS-Bot: Empowering Remote Sensing with VGI-Enhanced Large Multimodal Language Model](https://mlsanity.com/paper/2402.02544/) (2024)
- [SkyScript: A Large and Semantically Diverse Vision-Language Dataset for Remote Sensing](https://mlsanity.com/paper/2312.12856/) (2023)
- [RSVG: Exploring Data and Models for Visual Grounding on Remote Sensing Data](https://mlsanity.com/paper/2210.12634/) (2022)
- [Remote Sensing Vision-Language Foundation Models without Annotations via Ground Remote Alignment](https://mlsanity.com/paper/2312.06960/) (2023)
- [Charting New Territories: Exploring the Geographic and Geospatial Capabilities of Multimodal LLMs](https://mlsanity.com/paper/2311.14656/) (2023)
- [xView3-SAR: Detecting Dark Fishing Activity Using Synthetic Aperture Radar Imagery](https://mlsanity.com/paper/2206.00897/) (2022)
- [Fine-Tuning Language Models Using Formal Methods Feedback](https://mlsanity.com/paper/2310.18239/) (2023)
- [Stable Diffusion For Aerial Object Detection](https://mlsanity.com/paper/2311.12345/) (2023)
- [Rapid building damage assessment workflow: An implementation for the 2023 Rolling Fork, Mississippi tornado event](https://mlsanity.com/paper/2306.12589/) (2023)

## Built on by (most cited first, 30 of 30)

- [Multi-Agent Geospatial Copilots for Remote Sensing Workflows](https://mlsanity.com/paper/2501.16254/) (2025)
- [An LLM Agent for Automatic Geospatial Data Analysis](https://mlsanity.com/paper/2410.18792/) (2024)
- [An LLM-Tool Compiler for Fused Parallel Function Calling](https://mlsanity.com/paper/2405.17438/) (2024)
- [GIScience in the Era of Artificial Intelligence: A Research Agenda Towards Autonomous GIS](https://mlsanity.com/paper/2503.23633/) (2025)
- [GeckOpt: LLM System Efficiency via Intent-Based Tool Selection](https://mlsanity.com/paper/2404.15804/) (2024)
- [Towards Interactive Global Geolocation Assistant](https://mlsanity.com/paper/2412.08907/) (2024)
- [Geo-OLM: Enabling Sustainable Earth Observation Studies with Cost-Efficient Open Language Models & State-Driven Workflows](https://mlsanity.com/paper/2504.04319/) (2025)
- [ShapefileGPT: A Multi-Agent Large Language Model Framework for Automated Shapefile Processing](https://mlsanity.com/paper/2410.12376/) (2024)
- [CORE: Full-Path Evaluation of LLM Agents Beyond Final State](https://mlsanity.com/paper/2509.20998/) (2025)
- [GeoFlow: Agentic Workflow Automation for Geospatial Tasks](https://mlsanity.com/paper/2508.04719/) (2025)
- [LLM-dCache: Improving Tool-Augmented LLMs with GPT-Driven Localized Data Caching](https://mlsanity.com/paper/2406.06799/) (2024)
- [PowerChain: A Verifiable Agentic AI System for Automating Distribution Grid Analyses](https://mlsanity.com/paper/2508.17094/) (2025)
- [Agentic AI in Remote Sensing: Foundations, Taxonomy, and Emerging Systems](https://mlsanity.com/paper/2601.01891/) (2026)
- [Can LLM Agents Respond to Disasters? Benchmarking Heterogeneous Geospatial Reasoning in Emergency Operations](https://mlsanity.com/paper/2605.11633/) (2026)
- [Unlearning Climate Misinformation in Large Language Models](https://mlsanity.com/paper/2405.19563/) (2024)
- [On-Air Deep Learning Integrated Semantic Inference Models for Enhanced Earth Observation Satellite Networks](https://mlsanity.com/paper/2409.15246/) (2024)
- [Urban Computing in the Era of Large Language Models](https://mlsanity.com/paper/2504.02009/) (2025)
- [Geospatial Mechanistic Interpretability of Large Language Models](https://mlsanity.com/paper/2505.03368/) (2025)
- [RingMoClaw: An Experience-Inspired Multi-Agent Framework for Self-Evolving Research in Remote Sensing](https://mlsanity.com/paper/2609.00814/) (2026)
- [Carbon-Aware Routing for Function Calling in Edge-Cloud LLM Systems](https://mlsanity.com/paper/2609.13559/) (2026)
- [Naiad: Novel Agentic Intelligent Autonomous System for Inland Water Monitoring](https://mlsanity.com/paper/2601.05256/) (2025)
- [GeoAI Agency Primitives](https://mlsanity.com/paper/2604.01869/) (2026)
- [SatBLIP: Context Understanding and Feature Identification from Satellite Imagery with Vision-Language Learning](https://mlsanity.com/paper/2604.14373/) (2026)
- [Open-SAT: LLM-Guided Query Embedding Refinement for Open-Vocabulary Object Retrieval in Satellite Imagery](https://mlsanity.com/paper/2605.05344/) (2026)
- [Multimodal Large Language Models for Remote Sensing Image Understanding: Domain-Specific or General-Purpose?](https://mlsanity.com/paper/2607.20284/) (2026)
- [Deploying and Evaluating a Smart-Agriculture Agentic Engine for Full-Season Soybean Farm Operations](https://mlsanity.com/paper/2609.00106/) (2026)
- [Earth-Agent-Pro: Towards Real-World Full-Chain Earth Observation with Agents](https://mlsanity.com/paper/2609.12533/) (2026)
- [RSure-Agent: Reliable Use of Tool Observations for Remote Sensing Agents](https://mlsanity.com/paper/2610.04836/) (2026)
- [Trust Before Autonomy: Rethinking Agentic Remote Sensing for Crisis Response](https://mlsanity.com/paper/2610.18789/) (2026)
- [The Last Remote Sensing Model Built by Humans? Casting Any Earth Observation Task as Recursive Self-Improvement](https://mlsanity.com/paper/2610.18845/) (2026)

## Abstract (arXiv)

Geospatial Copilots unlock unprecedented potential for performing Earth Observation (EO) applications through natural language instructions. However, existing agents rely on overly simplified single tasks and template-based prompts, creating a disconnect with real-world scenarios. In this work, we present GeoLLM-Engine, an environment for tool-augmented agents with intricate tasks routinely executed by analysts on remote sensing platforms. We enrich our environment with geospatial API tools, dynamic maps/UIs, and external multimodal knowledge bases to properly gauge an agent's proficiency in interpreting realistic high-level natural language commands and its functional correctness in task completions. By alleviating overheads typically associated with human-in-the-loop benchmark curation, we harness our massively parallel engine across 100 GPT-4-Turbo nodes, scaling to over half a million diverse multi-tool tasks and across 1.1 million satellite images. By moving beyond traditional single-task image-caption paradigms, we investigate state-of-the-art agents and prompting techniques against long-horizon prompts.