# Assessing Large Language Models on Climate Information

arXiv: https://arxiv.org/abs/2310.02932 · HTML: https://mlsanity.com/paper/2310.02932/

## Key facts

- Cited by 23 papers in the map (data through 14 Oct 2026).
- Builds on 30 of its 104 references that are in the map.
- First posted on arXiv 4 Oct 2023.

## Builds on (most cited first, 30 of 30)

- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [Flamingo: a Visual Language Model for Few-Shot Learning](https://mlsanity.com/paper/2204.14198/) (2022)
- [Scaling Instruction-Finetuned Language Models](https://mlsanity.com/paper/2210.11416/) (2022)
- [Survey of Hallucination in Natural Language Generation](https://mlsanity.com/paper/2202.03629/) (2022)
- [Large Language Models Encode Clinical Knowledge](https://mlsanity.com/paper/2212.13138/) (2022)
- [Concrete Problems in AI Safety](https://mlsanity.com/paper/1606.06565/) (2016)
- [Ethical and social risks of harm from Language Models](https://mlsanity.com/paper/2112.04359/) (2021)
- [ChatGPT Outperforms Crowd-Workers for Text-Annotation Tasks](https://mlsanity.com/paper/2303.15056/) (2023)
- [PaLI: A Jointly-Scaled Multilingual Language-Image Model](https://mlsanity.com/paper/2209.06794/) (2022)
- [Can Large Language Models Be an Alternative to Human Evaluations?](https://mlsanity.com/paper/2305.01937/) (2023)
- [Image as a Foreign Language: BEiT Pretraining for All Vision and Vision-Language Tasks](https://mlsanity.com/paper/2208.10442/) (2022)
- [Scalable agent alignment via reward modeling: a research direction](https://mlsanity.com/paper/1811.07871/) (2018)
- [AI safety via debate](https://mlsanity.com/paper/1805.00899/) (2018)
- [Self-critiquing models for assisting human evaluators](https://mlsanity.com/paper/2206.05802/) (2022)
- [Evaluating Verifiability in Generative Search Engines](https://mlsanity.com/paper/2304.09848/) (2023)
- [Teaching language models to support answers with verified quotes](https://mlsanity.com/paper/2203.11147/) (2022)
- [Tackling Climate Change with Machine Learning](https://mlsanity.com/paper/1906.05433/) (2019)
- [Measuring Attribution in Natural Language Generation Models](https://mlsanity.com/paper/2112.12870/) (2021)
- [Measuring Progress on Scalable Oversight for Large Language Models](https://mlsanity.com/paper/2211.03540/) (2022)
- [CLIMATE-FEVER: A Dataset for Verification of Real-World Climate Claims](https://mlsanity.com/paper/2012.00614/) (2020)
- [Multilingual Universal Sentence Encoder for Semantic Retrieval](https://mlsanity.com/paper/1907.04307/) (2019)
- [Supervising strong learners by amplifying weak experts](https://mlsanity.com/paper/1810.08575/) (2018)
- [Attributed Question Answering: Evaluation and Modeling for Attributed Large Language Models](https://mlsanity.com/paper/2212.08037/) (2022)
- [A Critical Evaluation of Evaluations for Long-form Question Answering](https://mlsanity.com/paper/2305.18201/) (2023)
- [The Devil is in the Errors: Leveraging Large Language Models for Fine-grained Machine Translation Evaluation](https://mlsanity.com/paper/2308.07286/) (2023)
- [ClimateBert: A Pretrained Language Model for Climate-Related Text](https://mlsanity.com/paper/2110.12010/) (2021)
- [Evaluating Attribution in Dialogue Systems: The BEGIN Benchmark](https://mlsanity.com/paper/2105.00071/) (2021)
- [The Dangers of trusting Stochastic Parrots: Faithfulness and Trust in Open-domain Conversational Question Answering](https://mlsanity.com/paper/2305.16519/) (2023)
- [ClimaText: A Dataset for Climate Change Topic Detection](https://mlsanity.com/paper/2012.00483/) (2020)

## Built on by (most cited first, 23 of 23)

- [How Susceptible are LLMs to Influence in Prompts?](https://mlsanity.com/paper/2408.11865/) (2024)
- [ClimaQA: An Automated Evaluation Framework for Climate Question Answering Models](https://mlsanity.com/paper/2410.16701/) (2024)
- [From Text to Transformation: A Comprehensive Review of Large Language Models' Versatility](https://mlsanity.com/paper/2402.16142/) (2024)
- [WildfireGPT: Tailored Large Language Model for Wildfire Analysis](https://mlsanity.com/paper/2402.07877/) (2024)
- [ClimRetrieve: A Benchmarking Dataset for Information Retrieval from Corporate Climate Disclosures](https://mlsanity.com/paper/2406.09818/) (2024)
- [ClimateIQA: A New Dataset and Benchmark to Advance Vision-Language Models in Meteorology Anomalies Analysis](https://mlsanity.com/paper/2406.09838/) (2024)
- [Don't Think Twice! Over-Reasoning Impairs Confidence Calibration](https://mlsanity.com/paper/2508.15050/) (2025)
- [A RAG-Based Multi-Agent LLM System for Natural Hazard Resilience and Adaptation](https://mlsanity.com/paper/2504.17200/) (2025)
- [Automated Fact-Checking of Climate Change Claims with Large Language Models](https://mlsanity.com/paper/2401.12566/) (2024)
- [What can large language models do for sustainable food?](https://mlsanity.com/paper/2503.04734/) (2025)
- [Assessing Generative Language Models in Classification Tasks: Performance and Self-Evaluation Capabilities in the Environmental and Climate Change Domain](https://mlsanity.com/paper/2408.17362/) (2024)
- [ForesightSafety Bench: A Frontier Risk Evaluation and Governance Framework towards Safe AI](https://mlsanity.com/paper/2602.14135/) (2026)
- [CliME: Evaluating Multimodal Climate Discourse on Social Media and the Climate Alignment Quotient (CAQ)](https://mlsanity.com/paper/2504.03906/) (2025)
- [Generative Debunking of Climate Misinformation](https://mlsanity.com/paper/2407.05599/) (2024)
- [Judging It, Washing It: Scoring and Greenwashing Corporate Climate Disclosures using Large Language Models](https://mlsanity.com/paper/2502.15094/) (2025)
- [GeoGrid-Bench: Can Foundation Models Understand Multimodal Gridded Geo-Spatial Data?](https://mlsanity.com/paper/2505.10714/) (2025)
- [GCA Framework: A GCC Countries-Grounded Dataset and Agentic Pipeline for Climate Decision Support](https://mlsanity.com/paper/2604.12306/) (2026)
- [ClimaEmpact: Domain-Aligned Small Language Models and Datasets for Extreme Weather Analytics](https://mlsanity.com/paper/2504.19066/) (2025)
- [CLINB: A Climate Intelligence Benchmark for Foundational Models](https://mlsanity.com/paper/2511.11597/) (2025)
- [LLM Benchmark-User Need Misalignment for Climate Change](https://mlsanity.com/paper/2603.26106/) (2026)
- [MMClima: A Framework for Multimodal Climate Science Data and Evaluation](https://mlsanity.com/paper/2606.10194/) (2026)
- [Execution-grounded evaluation reveals hidden failures in language-model calculations for environmental science](https://mlsanity.com/paper/2608.18726/) (2026)
- [Eating for a Sustainable Planet: Personalized Sustainable Diet Recommendation via Constraint-Aware Decision-Making Modeling](https://mlsanity.com/paper/2608.24274/) (2026)

## Abstract (arXiv)

As Large Language Models (LLMs) rise in popularity, it is necessary to assess their capability in critically relevant domains. We present a comprehensive evaluation framework, grounded in science communication research, to assess LLM responses to questions about climate change. Our framework emphasizes both presentational and epistemological adequacy, offering a fine-grained analysis of LLM generations spanning 8 dimensions and 30 issues. Our evaluation task is a real-world example of a growing number of challenging problems where AI can complement and lift human performance. We introduce a novel protocol for scalable oversight that relies on AI Assistance and raters with relevant education. We evaluate several recent LLMs on a set of diverse climate questions. Our results point to a significant gap between surface and epistemological qualities of LLMs in the realm of climate communication.