# WXImpactBench: A Disruptive Weather Impact Understanding Benchmark for Evaluating Large Language Models

arXiv: https://arxiv.org/abs/2505.20249 · HTML: https://mlsanity.com/paper/2505.20249/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 22 of its 78 references that are in the map.
- First posted on arXiv 26 May 2025.

## Builds on (most cited first, 22 of 22)

- [The Llama 3 Herd of Models](https://mlsanity.com/paper/2407.21783/) (2024)
- [Scaling Laws for Neural Language Models](https://mlsanity.com/paper/2001.08361/) (2020)
- [Gemma 2: Improving Open Language Models at a Practical Size](https://mlsanity.com/paper/2408.00118/) (2024)
- [Harnessing the Power of LLMs in Practice: A Survey on ChatGPT and Beyond](https://mlsanity.com/paper/2304.13712/) (2023)
- [Retrieval-Augmented Generation for AI-Generated Content: A Survey](https://mlsanity.com/paper/2402.19473/) (2024)
- [WeatherBench: A benchmark dataset for data-driven weather forecasting](https://mlsanity.com/paper/2002.00469/) (2020)
- [WeatherBench 2: A benchmark for the next generation of data-driven global weather models](https://mlsanity.com/paper/2308.15560/) (2023)
- [Don't Make Your LLM an Evaluation Benchmark Cheater](https://mlsanity.com/paper/2311.01964/) (2023)
- [Large Language Models Know Your Contextual Search Intent: A Prompting Framework for Conversational Search](https://mlsanity.com/paper/2303.06573/) (2023)
- [ConvGQR: Generative Query Reformulation for Conversational Search](https://mlsanity.com/paper/2305.15645/) (2023)
- [Twitter as a Lifeline: Human-annotated Twitter Corpora for NLP of Crisis-related Messages](https://mlsanity.com/paper/1605.05894/) (2016)
- [RATT: A Thought Structure for Coherent and Correct LLM Reasoning](https://mlsanity.com/paper/2406.02746/) (2024)
- [CHIQ: Contextual History Enhancement for Improving Query Rewriting in Conversational Search](https://mlsanity.com/paper/2406.05013/) (2024)
- [Entropy-based Exploration Conduction for Multi-step Reasoning](https://mlsanity.com/paper/2503.15848/) (2025)
- [Bidirectional Encoder Representations from Transformers (BERT): A sentiment analysis odyssey](https://mlsanity.com/paper/2007.01127/) (2020)
- [Ranked List Truncation for Large Language Model-based Re-Ranking](https://mlsanity.com/paper/2404.18185/) (2024)
- [CLLMate: A Multimodal Benchmark for Weather and Climate Events Forecasting](https://mlsanity.com/paper/2409.19058/) (2024)
- [WildfireGPT: Tailored Large Language Model for Wildfire Analysis](https://mlsanity.com/paper/2402.07877/) (2024)
- [How to Leverage Personal Textual Knowledge for Personalized Conversational Information Retrieval](https://mlsanity.com/paper/2407.16192/) (2024)
- [OmniGeo: Towards a Multimodal Large Language Models for Geospatial Artificial Intelligence](https://mlsanity.com/paper/2503.16326/) (2025)
- [Information Retrieval for Climate Impact](https://mlsanity.com/paper/2504.01162/) (2025)
- [LACLICHEV: Exploring the History of Climate Change in Latin America within Newspapers Digital Collections](https://mlsanity.com/paper/2105.00792/) (2021)

## Abstract (arXiv)

Climate change adaptation requires the understanding of disruptive weather impacts on society, where large language models (LLMs) might be applicable. However, their effectiveness is under-explored due to the difficulty of high-quality corpus collection and the lack of available benchmarks. The climate-related events stored in regional newspapers record how communities adapted and recovered from disasters. However, the processing of the original corpus is non-trivial. In this study, we first develop a disruptive weather impact dataset with a four-stage well-crafted construction pipeline. Then, we propose WXImpactBench, the first benchmark for evaluating the capacity of LLMs on disruptive weather impacts. The benchmark involves two evaluation tasks, multi-label classification and ranking-based question answering. Extensive experiments on evaluating a set of LLMs provide first-hand analysis of the challenges in developing disruptive weather impact understanding and climate change adaptation systems. The constructed dataset and the code for the evaluation framework are available to help society protect against vulnerabilities from disasters.