# ClimateX: Do LLMs Accurately Assess Human Expert Confidence in Climate Statements?

arXiv: https://arxiv.org/abs/2311.17107 · HTML: https://mlsanity.com/paper/2311.17107/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 5 of its 18 references that are in the map.
- First posted on arXiv 28 Nov 2023.

## Builds on (most cited first, 5 of 5)

- [FEVER: a large-scale dataset for Fact Extraction and VERification](https://mlsanity.com/paper/1803.05355/) (2018)
- [Teaching Models to Express Their Uncertainty in Words](https://mlsanity.com/paper/2205.14334/) (2022)
- [Demonstrate-Search-Predict: Composing retrieval and language models for knowledge-intensive NLP](https://mlsanity.com/paper/2212.14024/) (2022)
- [ReCOGS: How Incidental Details of a Logical Form Overshadow an Evaluation of Semantic Interpretation](https://mlsanity.com/paper/2303.13716/) (2023)
- [ClimaText: A Dataset for Climate Change Topic Detection](https://mlsanity.com/paper/2012.00483/) (2020)

## Built on by (most cited first, 3 of 3)

- [Don't Think Twice! Over-Reasoning Impairs Confidence Calibration](https://mlsanity.com/paper/2508.15050/) (2025)
- [EAIRA: Establishing a Methodology for Evaluating AI Models as Scientific Research Assistants](https://mlsanity.com/paper/2502.20309/) (2025)
- [Assessing the Effectiveness of GPT-4o in Climate Change Evidence Synthesis and Systematic Assessments: Preliminary Insights](https://mlsanity.com/paper/2407.12826/) (2024)

## Abstract (arXiv)

Evaluating the accuracy of outputs generated by Large Language Models (LLMs) is especially important in the climate science and policy domain. We introduce the Expert Confidence in Climate Statements (ClimateX) dataset, a novel, curated, expert-labeled dataset consisting of 8094 climate statements collected from the latest Intergovernmental Panel on Climate Change (IPCC) reports, labeled with their associated confidence levels. Using this dataset, we show that recent LLMs can classify human expert confidence in climate-related statements, especially in a few-shot learning setting, but with limited (up to 47%) accuracy. Overall, models exhibit consistent and significant over-confidence on low and medium confidence statements. We highlight implications of our results for climate communication, LLMs evaluation strategies, and the use of LLMs in information retrieval systems.