# Evaluating Tool-Augmented Agents in Remote Sensing Platforms

arXiv: https://arxiv.org/abs/2405.00709 · HTML: https://mlsanity.com/paper/2405.00709/

## Key facts

- Cited by 17 papers in the map (data through 14 Oct 2026).
- Placed in Multimodal and embodied agents, one of 16 directions in LLM agents and tool use.
- Builds on 6 of its 18 references that are in the map.
- First posted on arXiv 23 Apr 2024.

## Placed in

- LLM agents and tool use / [Multimodal and embodied agents](https://mlsanity.com/agents/multimodal-and-embodied-agents/)

## Builds on (most cited first, 6 of 6)

- [MiniGPT-v2: large language model as a unified interface for vision-language multi-task learning](https://mlsanity.com/paper/2310.09478/) (2023)
- [ToolQA: A Dataset for LLM Question Answering with External Tools](https://mlsanity.com/paper/2306.13304/) (2023)
- [Object Detection in Aerial Images: A Large-Scale Benchmark and Challenges](https://mlsanity.com/paper/2102.12219/) (2021)
- [xView: Objects in Context in Overhead Imagery](https://mlsanity.com/paper/1802.07856/) (2018)
- [xView3-SAR: Detecting Dark Fishing Activity Using Synthetic Aperture Radar Imagery](https://mlsanity.com/paper/2206.00897/) (2022)
- [Stable Diffusion For Aerial Object Detection](https://mlsanity.com/paper/2311.12345/) (2023)

## Built on by (most cited first, 17 of 17)

- [Tool Learning with Large Language Models: A Survey](https://mlsanity.com/paper/2405.17935/) (2024)
- [Multi-Agent Geospatial Copilots for Remote Sensing Workflows](https://mlsanity.com/paper/2501.16254/) (2025)
- [Butterfly Effects in Toolchains: A Comprehensive Analysis of Failed Parameter Filling in LLM Tool-Agent Systems](https://mlsanity.com/paper/2507.15296/) (2025)
- [An LLM-Tool Compiler for Fused Parallel Function Calling](https://mlsanity.com/paper/2405.17438/) (2024)
- [GeckOpt: LLM System Efficiency via Intent-Based Tool Selection](https://mlsanity.com/paper/2404.15804/) (2024)
- [Geo-OLM: Enabling Sustainable Earth Observation Studies with Cost-Efficient Open Language Models & State-Driven Workflows](https://mlsanity.com/paper/2504.04319/) (2025)
- [Evaluating Zero-Shot GPT-4V Performance on 3D Visual Question Answering Benchmarks](https://mlsanity.com/paper/2405.18831/) (2024)
- [Build Your Personalized Research Group: A Multiagent Framework for Continual and Interactive Science Automation](https://mlsanity.com/paper/2510.15624/) (2025)
- [LLM-dCache: Improving Tool-Augmented LLMs with GPT-Driven Localized Data Caching](https://mlsanity.com/paper/2406.06799/) (2024)
- [RS-Claw: Progressive Active Tool Exploration via Hierarchical Skill Trees for Remote Sensing Agents](https://mlsanity.com/paper/2605.13391/) (2026)
- [Agentic AI for Remote Sensing: Technical Challenges and Research Directions](https://mlsanity.com/paper/2604.24919/) (2026)
- [Less is More: Optimizing Function Calling for LLM Execution on Edge Devices](https://mlsanity.com/paper/2411.15399/) (2024)
- [Remote Sensing Image Intelligent Interpretation with the Language-Centered Perspective: Principles, Methods and Challenges](https://mlsanity.com/paper/2508.06832/) (2025)
- [Enhancing Tool Calling in LLMs with the International Tool Calling Dataset](https://mlsanity.com/paper/2603.05515/) (2026)
- [GeoDisaster: Benchmarking Orchestrated Agents for Operational Disaster Geo-Intelligence](https://mlsanity.com/paper/2606.17246/) (2026)
- [Multimodal Large Language Models for Remote Sensing Image Understanding: Domain-Specific or General-Purpose?](https://mlsanity.com/paper/2607.20284/) (2026)
- [Agentic Multimodal Models for Environmental Hyperspectral Unmixing](https://mlsanity.com/paper/2609.01289/) (2026)

## Abstract (arXiv)

Tool-augmented Large Language Models (LLMs) have shown impressive capabilities in remote sensing (RS) applications. However, existing benchmarks assume question-answering input templates over predefined image-text data pairs. These standalone instructions neglect the intricacies of realistic user-grounded tasks. Consider a geospatial analyst: they zoom in a map area, they draw a region over which to collect satellite imagery, and they succinctly ask "Detect all objects here". Where is `here`, if it is not explicitly hardcoded in the image-text template, but instead is implied by the system state, e.g., the live map positioning? To bridge this gap, we present GeoLLM-QA, a benchmark designed to capture long sequences of verbal, visual, and click-based actions on a real UI platform. Through in-depth evaluation of state-of-the-art LLMs over a diverse set of 1,000 tasks, we offer insights towards stronger agents for RS applications.