mlsanityData to 7 Oct 2026

Towards LLM Agents for Earth Observation

arXiv 2504.12110 ↗First posted 16 Apr 2025cs.AIcited by 13

Not on the map. Why?

Abstract (arXiv)

Earth Observation (EO) provides critical planetary data for environmental monitoring, disaster management, climate science, and other scientific domains. In this work we ask: Are AI systems ready for reliable Earth Observation? To answer this, we introduce UnivEARTH, a coding benchmark of 408 yes/no questions from NASA Earth Observatory articles across 7 various topics and over 15 satellite instruments and sources. Using Google Earth Engine API as a tool in a zero-shot setup, LLM agents achieve an accuracy of 40.0% where the code fails to run over 44% of the time. To better understand LLM agent behavior, we also analyze the impact of using the JavaScript API versus Python and the effect of providing documentation. Furthermore, we find that using a Reflexion framework significantly reduces errors: Claude-4.5-Sonnet, Gemini-2.5-Pro, and GPT-5 accuracies rise to around 60%. However, these results remain only marginally above random chance. Taken together, our findings identify significant challenges to be solved before AI agents can automate earth observation, and suggest paths forward.

Key facts. Cited by 13 papers in the map (data through 7 Oct 2026). Builds on 33 of its 85 references that are in the map. First posted on arXiv 16 Apr 2025.

Builds on

33 of 85 references.

References · 33

  1. 2024The Llama 3 Herd of Modelscited by 14,192
  2. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  3. 2020Measuring Massive Multitask Language Understandingcited by 7,768
  4. 2024Qwen2.5 Technical Reportcited by 4,713
  5. 2024GPT-4o System Cardcited by 4,467
  6. 2023Reflexion: Language Agents with Verbal Reinforcement Learningcited by 4,064
  7. 2023SWE-bench: Can Language Models Resolve Real-World GitHub Issues?cited by 3,193
  8. 2024LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Codecited by 1,906
  9. 2018Know What You Don't Know: Unanswerable Questions for SQuADcited by 1,538
  10. 2024Qwen2.5-Coder Technical Reportcited by 1,151
  11. 2018Spider: A Large-Scale Human-Labeled Dataset for Complex and Cross-Domain Semantic Parsing and Text-to-SQL Taskcited by 1,120
  12. 2023Can LLM Already Serve as A Database Interface? A BIg Bench for Large-Scale Database Grounded Text-to-SQLscited by 667
21 more
  1. 2023ChemCrow: Augmenting large-language models with chemistry toolscited by 609
  2. 2020PathVQA: 30000+ Questions for Medical Visual Question Answeringcited by 467
  3. 2023SciBench: Evaluating College-Level Scientific Problem-Solving Abilities of Large Language Modelscited by 246
  4. 2024Spider 2.0: Evaluating Language Models on Real-World Enterprise Text-to-SQL Workflowscited by 223
  5. 2024SWE-bench Multimodal: Do AI Systems Generalize to Visual Software Domains?cited by 196
  6. 2024LAB-Bench: Measuring Capabilities of Language Models for Biology Researchcited by 138
  7. 2024SWT-Bench: Testing and Validating Real-World Bug-Fixes with Code Agentscited by 129
  8. 2024Change-Agent: Towards Interactive Comprehensive Remote Sensing Change Interpretation and Analysiscited by 60
  9. 2023T-SciQ: Teaching Multimodal Chain-of-Thought Reasoning via Mixed Large Language Model Signals for Science Question Answeringcited by 59
  10. 2022(QA)2: Question Answering with Questionable Assumptionscited by 51
  11. 2023EarthVQA: Towards Queryable Earth via Relational Reasoning-Based Remote Sensing Visual Question Answeringcited by 51
  12. 2024GEOBench-VLM: Benchmarking Vision-Language Models for Geospatial Taskscited by 50
  13. 2024RS-Agent: Automating Remote Sensing Tasks through Intelligent Agentcited by 46
  14. 2021Text-to-SQL in the Wild: A Naturally-Occurring Dataset Based on Stack Exchange Datacited by 43
  15. 2024OSCAR: Operating System Control via State-Aware Reasoning and Re-Planningcited by 37
  16. 2024SceMQA: A Scientific College Entrance Level Multimodal Question Answering Benchmarkcited by 35
  17. 2024Probing the limitations of multimodal language models for chemistry and materials researchcited by 31
  18. 2024MMSci: A Dataset for Graduate-Level Multi-Discipline Multimodal Scientific Understandingcited by 23
  19. 2025MDCrow: Automating Molecular Dynamics Workflows with Large Language Modelscited by 22
  20. 2024Show Me What and Where has Changed? Question Answering and Grounding for Remote Sensing Change Detectioncited by 15
  21. 2024SwiftDossier: Tailored Automatic Dossier for Drug Discovery with LLMs and Agentscited by 3

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 3; 2026: 10.

13 papers.

Sort:

By area:

  1. 2025Earth-Agent: Unlocking the Full Landscape of Earth Observation with Agentscited by 42
  2. 2025ThinkGeo: Evaluating Tool-Augmented Agents for Remote Sensing Taskscited by 41
  3. 2026Can LLM Agents Respond to Disasters? Benchmarking Heterogeneous Geospatial Reasoning in Emergency Operationscited by 4
  4. 2026TerraBench: Can Agents Reason Over Heterogeneous Earth-System Data?cited by 4
  5. 2026IC-EO: Interpretable Code-based assistant for Earth Observationcited by 2
  6. 2026TerraLogic: A Benchmark for Hierarchical Geospatial Reasoning in Earth Observationcited by 2
  7. 2026Agentic AI for Remote Sensing: Technical Challenges and Research Directionscited by 1
  8. 2026EO-Gym: A Multimodal, Interactive Environment for Earth Observation Agentscited by 1
  9. 2025Naiad: Novel Agentic Intelligent Autonomous System for Inland Water Monitoringcited by 0
  10. 2026Geo-Expert: Towards Expert-Level Geological Reasoning via Parameter-Efficient Fine-Tuningcited by 0
  11. 2026GeoNatureAgent Benchmark: Benchmarking LLM Agents for Environmental Geospatial Analysis Across Frontier and Open-Weight Foundation Modelscited by 0
  12. 2026Earth-Agent-Pro: Towards Real-World Full-Chain Earth Observation with Agentscited by 0
1 more
  1. 2026From Pixels to Policy: A Multi-Agent System for Intervention and Geo-Spatial Decision Supportcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026EO-Agents: A Three-Agent LLM Pipeline for Earth Observation Hypothesis Generation
  2. 2026OpenEarth-Agent: From Tool Calling to Tool Creation for Open-Environment Earth Observation
  3. 2026EVE: A Domain-Specific LLM Framework for Earth Intelligence
  4. 2022EarthNets: Empowering AI in Earth Observation
  5. 2026Risk-Aware LLM Agents for Geospatial Data Retrieval: Design and Preliminary Adversarial Evaluation
  6. 2025Earth AI: Unlocking Geospatial Insights with Foundation Models and Cross-Modal Reasoning
  7. 2025Geo-OLM: Enabling Sustainable Earth Observation Studies with Cost-Efficient Open Language Models & State-Driven Workflows
  8. 2026Bringing Agentic Search to Earth Observation Data Discovery