# AutoGEEval: A Multimodal and Automated Framework for Geospatial Code Generation on GEE with Large Language Models

arXiv: https://arxiv.org/abs/2505.12900 · HTML: https://mlsanity.com/paper/2505.12900/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 22 of its 51 references that are in the map.
- First posted on arXiv 19 May 2025.

## Builds on (most cited first, 22 of 22)

- [LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Code](https://mlsanity.com/paper/2403.07974/) (2024)
- [CodeBERT: A Pre-Trained Model for Programming and Natural Languages](https://mlsanity.com/paper/2002.08155/) (2020)
- [Qwen2.5-Coder Technical Report](https://mlsanity.com/paper/2409.12186/) (2024)
- [DeepSeek-Coder: When the Large Language Model Meets Programming -- The Rise of Code Intelligence](https://mlsanity.com/paper/2401.14196/) (2024)
- [Seq2SQL: Generating Structured Queries from Natural Language using Reinforcement Learning](https://mlsanity.com/paper/1709.00103/) (2017)
- [code2seq: Generating Sequences from Structured Representations of Code](https://mlsanity.com/paper/1808.01400/) (2018)
- [GitHub Copilot AI pair programmer: Asset or Liability?](https://mlsanity.com/paper/2206.15331/) (2022)
- [DeepCoder: Learning to Write Programs](https://mlsanity.com/paper/1611.01989/) (2016)
- [Large Language Models Meet NL2Code: A Survey](https://mlsanity.com/paper/2212.09420/) (2022)
- [Transformers in Remote Sensing: A Survey](https://mlsanity.com/paper/2209.01206/) (2022)
- [On the Effectiveness of Large Language Models in Domain-Specific Code Generation](https://mlsanity.com/paper/2312.01639/) (2023)
- [A Survey on Evaluating Large Language Models in Code Generation Tasks](https://mlsanity.com/paper/2408.16498/) (2024)
- [HumanEval Pro and MBPP Pro: Evaluating Large Language Models on Self-invoking Code Generation](https://mlsanity.com/paper/2412.21199/) (2024)
- [GIS Copilot: Towards an Autonomous GIS Agent for Spatial Analysis](https://mlsanity.com/paper/2411.03205/) (2024)
- [GeoCode-GPT: A Large Language Model for Geospatial Code Generation Tasks](https://mlsanity.com/paper/2410.17031/) (2024)
- [ShapefileGPT: A Multi-Agent Large Language Model Framework for Automated Shapefile Processing](https://mlsanity.com/paper/2410.12376/) (2024)
- [Evaluation of Code LLMs on Geospatial Code Generation](https://mlsanity.com/paper/2410.04617/) (2024)
- [HumanEval on Latest GPT Models -- 2024](https://mlsanity.com/paper/2402.14852/) (2024)
- [Can Large Language Models Generate Geospatial Code?](https://mlsanity.com/paper/2410.09738/) (2024)
- [Systematic Mapping Study of Template-based Code Generation](https://mlsanity.com/paper/1703.06353/) (2017)
- [Chain-of-Programming (CoP) : Empowering Large Language Models for Geospatial Code Generation](https://mlsanity.com/paper/2411.10753/) (2024)
- [GEE-OPs: An Operator Knowledge Base for Geospatial Code Generation on the Google Earth Engine Platform Powered by Large Language Models](https://mlsanity.com/paper/2412.05587/) (2024)

## Abstract (arXiv)

Geospatial code generation is emerging as a key direction in the integration of artificial intelligence and geoscientific analysis. However, there remains a lack of standardized tools for automatic evaluation in this domain. To address this gap, we propose AutoGEEval, the first multimodal, unit-level automated evaluation framework for geospatial code generation tasks on the Google Earth Engine (GEE) platform powered by large language models (LLMs). Built upon the GEE Python API, AutoGEEval establishes a benchmark suite (AutoGEEval-Bench) comprising 1325 test cases that span 26 GEE data types. The framework integrates both question generation and answer verification components to enable an end-to-end automated evaluation pipeline-from function invocation to execution validation. AutoGEEval supports multidimensional quantitative analysis of model outputs in terms of accuracy, resource consumption, execution efficiency, and error types. We evaluate 18 state-of-the-art LLMs-including general-purpose, reasoning-augmented, code-centric, and geoscience-specialized models-revealing their performance characteristics and potential optimization pathways in GEE code generation. This work provides a unified protocol and foundational resource for the development and assessment of geospatial code generation models, advancing the frontier of automated natural language to domain-specific code translation.