mlsanityData to 14 Oct 2026

Assessing Large Language Models on Climate Information

arXiv 2310.02932 ↗First posted 4 Oct 2023cs.CL cs.AI cs.CYcited by 23

Not on the map. Why?

Abstract (arXiv)

As Large Language Models (LLMs) rise in popularity, it is necessary to assess their capability in critically relevant domains. We present a comprehensive evaluation framework, grounded in science communication research, to assess LLM responses to questions about climate change. Our framework emphasizes both presentational and epistemological adequacy, offering a fine-grained analysis of LLM generations spanning 8 dimensions and 30 issues. Our evaluation task is a real-world example of a growing number of challenging problems where AI can complement and lift human performance. We introduce a novel protocol for scalable oversight that relies on AI Assistance and raters with relevant education. We evaluate several recent LLMs on a set of diverse climate questions. Our results point to a significant gap between surface and epistemological qualities of LLMs in the realm of climate communication.

Key facts. Cited by 23 papers in the map (data through 14 Oct 2026). Builds on 30 of its 104 references that are in the map. First posted on arXiv 4 Oct 2023.

Builds on

30 of 104 references.

References · 30

  1. 2020Language Models are Few-Shot Learnerscited by 32,419
  2. 2022Training language models to follow instructions with human feedbackcited by 15,971
  3. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,386
  4. 2022Scaling Instruction-Finetuned Language Modelscited by 3,024
  5. 2022Survey of Hallucination in Natural Language Generationcited by 2,075
  6. 2022Large Language Models Encode Clinical Knowledgecited by 1,830
  7. 2016Concrete Problems in AI Safetycited by 1,642
  8. 2021Ethical and social risks of harm from Language Modelscited by 966
  9. 2023ChatGPT Outperforms Crowd-Workers for Text-Annotation Taskscited by 870
  10. 2022PaLI: A Jointly-Scaled Multilingual Language-Image Modelcited by 766
  11. 2023Can Large Language Models Be an Alternative to Human Evaluations?cited by 734
  12. 2022Image as a Foreign Language: BEiT Pretraining for All Vision and Vision-Language Taskscited by 511
18 more
  1. 2018Scalable agent alignment via reward modeling: a research directioncited by 447
  2. 2018AI safety via debatecited by 400
  3. 2022Self-critiquing models for assisting human evaluatorscited by 380
  4. 2023Evaluating Verifiability in Generative Search Enginescited by 303
  5. 2022Teaching language models to support answers with verified quotescited by 292
  6. 2019Tackling Climate Change with Machine Learningcited by 265
  7. 2021Measuring Attribution in Natural Language Generation Modelscited by 256
  8. 2022Measuring Progress on Scalable Oversight for Large Language Modelscited by 225
  9. 2020CLIMATE-FEVER: A Dataset for Verification of Real-World Climate Claimscited by 196
  10. 2019Multilingual Universal Sentence Encoder for Semantic Retrievalcited by 193
  11. 2018Supervising strong learners by amplifying weak expertscited by 145
  12. 2022Attributed Question Answering: Evaluation and Modeling for Attributed Large Language Modelscited by 112
  13. 2023A Critical Evaluation of Evaluations for Long-form Question Answeringcited by 111
  14. 2023The Devil is in the Errors: Leveraging Large Language Models for Fine-grained Machine Translation Evaluationcited by 95
  15. 2021ClimateBert: A Pretrained Language Model for Climate-Related Textcited by 90
  16. 2021Evaluating Attribution in Dialogue Systems: The BEGIN Benchmarkcited by 77
  17. 2023The Dangers of trusting Stochastic Parrots: Faithfulness and Trust in Open-domain Conversational Question Answeringcited by 26
  18. 2020ClimaText: A Dataset for Climate Change Topic Detectioncited by 22

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 9; 2025: 8; 2026: 6.

23 papers.

Sort:

By area:

  1. 2024How Susceptible are LLMs to Influence in Prompts?cited by 33
  2. 2024ClimaQA: An Automated Evaluation Framework for Climate Question Answering Modelscited by 21
  3. 2024From Text to Transformation: A Comprehensive Review of Large Language Models' Versatilitycited by 19
  4. 2024WildfireGPT: Tailored Large Language Model for Wildfire Analysiscited by 15
  5. 2024ClimRetrieve: A Benchmarking Dataset for Information Retrieval from Corporate Climate Disclosurescited by 12
  6. 2024ClimateIQA: A New Dataset and Benchmark to Advance Vision-Language Models in Meteorology Anomalies Analysiscited by 10
  7. 2025Don't Think Twice! Over-Reasoning Impairs Confidence Calibrationcited by 6
  8. 2025A RAG-Based Multi-Agent LLM System for Natural Hazard Resilience and Adaptationcited by 5
  9. 2024Automated Fact-Checking of Climate Change Claims with Large Language Modelscited by 4
  10. 2025What can large language models do for sustainable food?cited by 4
  11. 2024Assessing Generative Language Models in Classification Tasks: Performance and Self-Evaluation Capabilities in the Environmental and Climate Change Domaincited by 3
  12. 2026ForesightSafety Bench: A Frontier Risk Evaluation and Governance Framework towards Safe AIcited by 3
11 more
  1. 2025CliME: Evaluating Multimodal Climate Discourse on Social Media and the Climate Alignment Quotient (CAQ)cited by 2
  2. 2024Generative Debunking of Climate Misinformationcited by 1
  3. 2025Judging It, Washing It: Scoring and Greenwashing Corporate Climate Disclosures using Large Language Modelscited by 1
  4. 2025GeoGrid-Bench: Can Foundation Models Understand Multimodal Gridded Geo-Spatial Data?cited by 1
  5. 2026GCA Framework: A GCC Countries-Grounded Dataset and Agentic Pipeline for Climate Decision Supportcited by 1
  6. 2025ClimaEmpact: Domain-Aligned Small Language Models and Datasets for Extreme Weather Analyticscited by 0
  7. 2025CLINB: A Climate Intelligence Benchmark for Foundational Modelscited by 0
  8. 2026LLM Benchmark-User Need Misalignment for Climate Changecited by 0
  9. 2026MMClima: A Framework for Multimodal Climate Science Data and Evaluationcited by 0
  10. 2026Execution-grounded evaluation reveals hidden failures in language-model calculations for environmental sciencecited by 0
  11. 2026Eating for a Sustainable Planet: Personalized Sustainable Diet Recommendation via Constraint-Aware Decision-Making Modelingcited by 0