# Assessing the Effectiveness of GPT-4o in Climate Change Evidence Synthesis and Systematic Assessments: Preliminary Insights

arXiv: https://arxiv.org/abs/2407.12826 · HTML: https://mlsanity.com/paper/2407.12826/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 9 of its 21 references that are in the map.
- First posted on arXiv 2 Jul 2024.

## Builds on (most cited first, 9 of 9)

- [Is GPT-3 a Good Data Annotator?](https://mlsanity.com/paper/2212.10450/) (2022)
- [AnnoLLM: Making Large Language Models to Be Better Crowdsourced Annotators](https://mlsanity.com/paper/2303.16854/) (2023)
- [Stance Detection with Bidirectional Conditional Encoding](https://mlsanity.com/paper/1606.05464/) (2016)
- [LLMs Accelerate Annotation for Medical Information Extraction](https://mlsanity.com/paper/2312.02296/) (2023)
- [Growth rates of modern science: A latent piecewise growth curve approach to model publication numbers from established and new literature databases](https://mlsanity.com/paper/2012.07675/) (2020)
- [ClimateGPT: Towards AI Synthesizing Interdisciplinary Research on Climate Change](https://mlsanity.com/paper/2401.09646/) (2024)
- [Enhancing Large Language Models with Climate Resources](https://mlsanity.com/paper/2304.00116/) (2023)
- [Can Large Language Models Discern Evidence for Scientific Hypotheses? Case Studies in the Social Sciences](https://mlsanity.com/paper/2309.06578/) (2023)
- [ClimateX: Do LLMs Accurately Assess Human Expert Confidence in Climate Statements?](https://mlsanity.com/paper/2311.17107/) (2023)

## Built on by (most cited first, 2 of 2)

- [3DAxisPrompt: Promoting the 3D Grounding and Reasoning in GPT-4o](https://mlsanity.com/paper/2503.13185/) (2025)
- [CliME: Evaluating Multimodal Climate Discourse on Social Media and the Climate Alignment Quotient (CAQ)](https://mlsanity.com/paper/2504.03906/) (2025)

## Abstract (arXiv)

In this research short, we examine the potential of using GPT-4o, a state-of-the-art large language model (LLM) to undertake evidence synthesis and systematic assessment tasks. Traditional workflows for such tasks involve large groups of domain experts who manually review and synthesize vast amounts of literature. The exponential growth of scientific literature and recent advances in LLMs provide an opportunity to complementing these traditional workflows with new age tools. We assess the efficacy of GPT-4o to do these tasks on a sample from the dataset created by the Global Adaptation Mapping Initiative (GAMI) where we check the accuracy of climate change adaptation related feature extraction from the scientific literature across three levels of expertise. Our results indicate that while GPT-4o can achieve high accuracy in low-expertise tasks like geographic location identification, their performance in intermediate and high-expertise tasks, such as stakeholder identification and assessment of depth of the adaptation response, is less reliable. The findings motivate the need for designing assessment workflows that utilize the strengths of models like GPT-4o while also providing refinements to improve their performance on these tasks.