# Execution-grounded evaluation reveals hidden failures in language-model calculations for environmental science

arXiv: https://arxiv.org/abs/2608.18726 · HTML: https://mlsanity.com/paper/2608.18726/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 18 of its 59 references that are in the map.
- First posted on arXiv 19 Aug 2026.

## Builds on (most cited first, 18 of 18)

- [PAL: Program-aided Language Models](https://mlsanity.com/paper/2211.10435/) (2022)
- [GSM-Symbolic: Understanding the Limitations of Mathematical Reasoning in Large Language Models](https://mlsanity.com/paper/2410.05229/) (2024)
- [Large Language Models Are Not Robust Multiple Choice Selectors](https://mlsanity.com/paper/2309.03882/) (2023)
- [How Hungry is AI? Benchmarking Energy, Water, and Carbon Footprint of LLM Inference](https://mlsanity.com/paper/2505.09598/) (2025)
- [Can multiple-choice questions really be useful in detecting the abilities of LLMs?](https://mlsanity.com/paper/2403.17752/) (2024)
- [ClimateGPT: Towards AI Synthesizing Interdisciplinary Research on Climate Change](https://mlsanity.com/paper/2401.09646/) (2024)
- [FrontierScience: Evaluating AI's Ability to Perform Expert-Level Scientific Tasks](https://mlsanity.com/paper/2601.21165/) (2026)
- [Changing Answer Order Can Decrease MMLU Accuracy](https://mlsanity.com/paper/2406.19470/) (2024)
- [GeoGalactica: A Scientific Large Language Model in Geoscience](https://mlsanity.com/paper/2401.00434/) (2023)
- [Assessing Large Language Models on Climate Information](https://mlsanity.com/paper/2310.02932/) (2023)
- [ClimateBERT-NetZero: Detecting and Assessing Net Zero and Reduction Targets](https://mlsanity.com/paper/2310.08096/) (2023)
- [Who Benchmarks the Benchmarks? Towards Comprehensive Evaluation of Commonsense Reasoning Benchmarks](https://mlsanity.com/paper/2504.07825/) (2025)
- [Towards Answering Climate Questionnaires from Unstructured Climate Reports](https://mlsanity.com/paper/2301.04253/) (2023)
- [Zephyrus: An Agentic Framework for Weather Science](https://mlsanity.com/paper/2510.04017/) (2025)
- [AtmosSci-Bench: Evaluating the Recent Advance of Large Language Model for Atmospheric Science](https://mlsanity.com/paper/2502.01159/) (2025)
- [EnviroExam: Benchmarking Environmental Science Knowledge of Large Language Models](https://mlsanity.com/paper/2405.11265/) (2024)
- [ClimAgent: LLM as Agents for Autonomous Open-ended Climate Science Analysis](https://mlsanity.com/paper/2604.16922/) (2026)
- [Composing Open-domain Vision with RAG for Ocean Monitoring and Conservation](https://mlsanity.com/paper/2412.02262/) (2024)

## Abstract (arXiv)

Large language models are increasingly used for quantitative work in the environmental sciences, yet existing evaluations score only final answers, leaving calculation process unobserved. Here we introduce AtmosCoder-Bench, an execution-grounded benchmark that makes the calculation process visible. Built through a transferable semi-automated pipeline (436 problems, 3,910 variants, 7,029 graded quantities), every problem is validated to be unambiguous and human-solvable, with uniquely verifiable answers. We find that (i) multiple-choice formats inflate measured accuracy by at least 12 percentage points; (ii) many failures arise not from missing knowledge but from models failing to apply known formulas and constraints consistently throughout multi-step computation; and (iii) even frontier models remain weak when task-specific conditions invalidate familiar methods, often reverting to canonical solution patterns rather than adapting methods to the relevant physical regime, leaving expert oversight essential.