mlsanityData to 7 Oct 2026

ClimaQA: An Automated Evaluation Framework for Climate Question Answering Models

arXiv 2410.16701 ↗First posted 22 Oct 2024cs.LGcited by 21

Not on the map. Why?

Abstract (arXiv)

The use of Large Language Models (LLMs) in climate science has recently gained significant attention. However, a critical issue remains: the lack of a comprehensive evaluation framework capable of assessing the quality and scientific validity of model outputs. To address this issue, we develop ClimaGen (Climate QA Generator), an adaptive learning framework that generates question-answer pairs from graduate textbooks with climate scientists in the loop. As a result, we present ClimaQA-Gold, an expert-annotated benchmark dataset alongside ClimaQA-Silver, a large-scale, comprehensive synthetic QA dataset for climate science. Finally, we develop evaluation strategies and compare different LLMs on our benchmarks. Our results offer novel insights into various approaches used to enhance knowledge of climate LLMs. The source code is publicly available at https://github.com/Rose-STL-Lab/genie-climaqa

Key facts. Cited by 21 papers in the map (data through 7 Oct 2026). Builds on 22 of its 59 references that are in the map. First posted on arXiv 22 Oct 2024.

Builds on

22 of 59 references.

References · 22

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2023GPT-4 Technical Reportcited by 18,132
  3. 2024The Llama 3 Herd of Modelscited by 14,192
  4. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  5. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  6. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,160
  7. 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,049
  8. 2019BERTScore: Evaluating Text Generation with BERTcited by 5,462
  9. 2024Gemini 1.5: Unlocking multimodal understanding across millions of tokens of contextcited by 3,175
  10. 2024Gemma 2: Improving Open Language Models at a Practical Sizecited by 1,928
  11. 2022Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question Answeringcited by 1,881
  12. 2024Mixtral of Expertscited by 1,785
10 more
  1. 2018Universal Sentence Encodercited by 485
  2. 2020CLIMATE-FEVER: A Dataset for Verification of Real-World Climate Claimscited by 195
  3. 2023LLMaAA: Making Large Language Models as Active Annotatorscited by 78
  4. 2023A Comparative Study of AI-Generated (GPT-4) and Human-crafted MCQs in Programming Educationcited by 34
  5. 2023Assessing Large Language Models on Climate Informationcited by 22
  6. 2024Are Machines Better at Complex Reasoning? Unveiling Human-Machine Inference Gaps in Entailment Verificationcited by 13
  7. 2021More Than Reading Comprehension: A Survey on Datasets and Metrics of Textual Question Answeringcited by 8
  8. 2021An Automated Multiple-Choice Question Generation Using Natural Language Processing Techniquescited by 6
  9. 2023Climate Change from Large Language Modelscited by 6
  10. 2024STAR: Constraint LoRA with Dynamic Active Learning for Data-Efficient Fine-Tuning of Large Language Modelscited by 5

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 6; 2026: 15.

21 papers.

Sort:

By area:

  1. 2025A Survey of Scientific Large Language Models: From Data Foundations to Agent Frontierscited by 34
  2. 2026OpenCompass: A Universal Evaluation Platform for Large Language Modelscited by 5
  3. 2025SimulRAG: Simulator-based RAG for Grounding LLMs in Long-form Scientific QAcited by 3
  4. 2025Mining Useful General Data for Low-Resource Domain Adaptationcited by 3
  5. 2025SciEvalKit: An Open-source Evaluation Toolkit for Scientific General Intelligencecited by 3
  6. 2026DataChef: Cooking Up Optimal Data Recipes for LLM Adaptation via Reinforcement Learningcited by 2
  7. 2026GCA Framework: A GCC Countries-Grounded Dataset and Agentic Pipeline for Climate Decision Supportcited by 1
  8. 2026ClimAgent: LLM as Agents for Autonomous Open-ended Climate Science Analysiscited by 1
  9. 2025WeatherArchive-Bench: Benchmarking Retrieval-Augmented Reasoning for Historical Weather Archivescited by 0
  10. 2025Sensorium Arc: AI Agent System for Oceanic Data Exploration and Interactive Eco-Artcited by 0
  11. 2026LLM Benchmark-User Need Misalignment for Climate Changecited by 0
  12. 2026K-MetBench: A Multi-Dimensional Benchmark for Fine-Grained Evaluation of Expert Reasoning, Locality, and Multimodality in Meteorologycited by 0
9 more
  1. 2026EvoSelect: Data-Efficient LLM Evolution for Targeted Task Adaptationcited by 0
  2. 2026Earth Science Foundation Models: From Perception to Reasoning and Discoverycited by 0
  3. 2026ReCrit: Transition-Aware Reinforcement Learning for Scientific Critic Reasoningcited by 0
  4. 2026Reasoning4Sciences: Bridging Reasoning Language Models to All Scientific Branchescited by 0
  5. 2026MMClima: A Framework for Multimodal Climate Science Data and Evaluationcited by 0
  6. 2026Hybrid NARX-LLM for Greenland Iceberg Discharge: Prompt-Driven Residual Correctioncited by 0
  7. 2026Climate-ModernBERT: Revisiting Corpus Composition for Domain-Adaptive Continued Pretrainingcited by 0
  8. 2026S3-Bench: Evaluating Speech Interaction Models as Scientific Voice Assistantscited by 0
  9. 2026Scientific Agents: Evaluating Profession-Specific System Prompts on Scientific Taskscited by 0