# CLLMate: A Multimodal Benchmark for Weather and Climate Events Forecasting

arXiv: https://arxiv.org/abs/2409.19058 · HTML: https://mlsanity.com/paper/2409.19058/

## Key facts

- Cited by 20 papers in the map (data through 14 Oct 2026).
- Builds on 24 of its 64 references that are in the map.
- First posted on arXiv 27 Sep 2024.

## Builds on (most cited first, 24 of 24)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Measuring Massive Multitask Language Understanding](https://mlsanity.com/paper/2009.03300/) (2020)
- [Large Language Models are Zero-Shot Reasoners](https://mlsanity.com/paper/2205.11916/) (2022)
- [Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolution](https://mlsanity.com/paper/2409.12191/) (2024)
- [InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Tasks](https://mlsanity.com/paper/2312.14238/) (2023)
- [MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGI](https://mlsanity.com/paper/2311.16502/) (2023)
- [DeepSeek-VL: Towards Real-World Vision-Language Understanding](https://mlsanity.com/paper/2403.05525/) (2024)
- [NExT-GPT: Any-to-Any Multimodal LLM](https://mlsanity.com/paper/2309.05519/) (2023)
- [Multimodal Learning with Transformers: A Survey](https://mlsanity.com/paper/2206.06488/) (2022)
- [InternLM-XComposer2: Mastering Free-form Text-Image Composition and Comprehension in Vision-Language Large Model](https://mlsanity.com/paper/2401.16420/) (2024)
- [AnyGPT: Unified Multimodal LLM with Discrete Sequence Modeling](https://mlsanity.com/paper/2402.12226/) (2024)
- [COLA: Contextualized Commonsense Causal Reasoning from the Causal Inference Perspective](https://mlsanity.com/paper/2305.05191/) (2023)
- [WeatherQA: Can Multimodal Language Models Reason about Severe Weather?](https://mlsanity.com/paper/2406.11217/) (2024)
- [Mesogeos: A multi-purpose dataset for data-driven wildfire modeling in the Mediterranean](https://mlsanity.com/paper/2306.05144/) (2023)
- [Digital Typhoon: Long-term Satellite Image Dataset for the Spatio-Temporal Modeling of Tropical Cyclones](https://mlsanity.com/paper/2311.02665/) (2023)
- [Are Large Language Models a Good Replacement of Taxonomies?](https://mlsanity.com/paper/2406.11131/) (2024)
- [AbsPyramid: Benchmarking the Abstraction Ability of Language Models with a Unified Entailment Graph](https://mlsanity.com/paper/2311.09174/) (2023)
- [SubeventWriter: Iterative Sub-event Sequence Generation with Coherence Controller](https://mlsanity.com/paper/2210.06694/) (2022)
- [Robustness of Structured Data Extraction from In-plane Rotated Documents using Multi-Modal Large Language Models (LLM)](https://mlsanity.com/paper/2406.10295/) (2024)
- [ConstraintChecker: A Plugin for Large Language Models to Reason on Commonsense Knowledge Bases](https://mlsanity.com/paper/2401.14003/) (2024)
- [Identifying robust communities and multi-community nodes by combining top-down and bottom-up approaches to clustering](https://mlsanity.com/paper/1501.04709/) (2015)

## Built on by (most cited first, 20 of 20)

- [A Survey of Scientific Large Language Models: From Data Foundations to Agent Frontiers](https://mlsanity.com/paper/2508.21148/) (2025)
- [Scientists' First Exam: Probing Cognitive Abilities of MLLM via Perception, Understanding, and Reasoning](https://mlsanity.com/paper/2506.10521/) (2025)
- [EF-LLM: Energy Forecasting LLM with AI-assisted Automation, Enhanced Sparse Prediction, Hallucination Detection](https://mlsanity.com/paper/2411.00852/) (2024)
- [KERAG: Knowledge-Enhanced Retrieval-Augmented Generation for Advanced Question Answering](https://mlsanity.com/paper/2509.04716/) (2025)
- [TerraBench: Can Agents Reason Over Heterogeneous Earth-System Data?](https://mlsanity.com/paper/2606.13148/) (2026)
- [MeteorPred: A Meteorological Multimodal Large Model and Dataset for Severe Weather Event Prediction](https://mlsanity.com/paper/2508.06859/) (2025)
- [Hierarchical AI-Meteorologist: LLM-Agent System for Multi-Scale and Explainable Weather Forecast Reporting](https://mlsanity.com/paper/2511.23387/) (2025)
- [AutoSAM: an Agentic Framework for Automating Input File Generation for the SAM Code with Multi-Modal Retrieval-Augmented Generation](https://mlsanity.com/paper/2603.24736/) (2026)
- [SIREN: Towards End-to-End Extreme-Weather Early Warning with Experience-Grounded LLM Agents](https://mlsanity.com/paper/2607.24588/) (2026)
- [WeatherSyn: An Instruction Tuning MLLM For Weather Forecasting Report Generation](https://mlsanity.com/paper/2605.07522/) (2026)
- [Not All Errors Are Equal: A Systematic Study of Error Propagation in Large Language Model Inference](https://mlsanity.com/paper/2606.02430/) (2026)
- [WXImpactBench: A Disruptive Weather Impact Understanding Benchmark for Evaluating Large Language Models](https://mlsanity.com/paper/2505.20249/) (2025)
- [A Modular LLM-Agent System for Transparent Multi-Parameter Weather Interpretation](https://mlsanity.com/paper/2512.11819/) (2025)
- [SolarGPT-QA: A Domain-Adaptive Large Language Model for Educational Question Answering in Space Weather and Heliophysics](https://mlsanity.com/paper/2601.12131/) (2026)
- [S2SServiceBench: A Multimodal Benchmark for Last-Mile S2S Climate Services](https://mlsanity.com/paper/2602.14017/) (2026)
- [Optimizing Multi-Agent Weather Captioning via Text Gradient Descent: A Training-Free Approach with Consensus-Aware Gradient Fusion](https://mlsanity.com/paper/2603.21673/) (2026)
- [SynopticBench: Evaluating Vision-Language Models on Generating Weather Forecast Discussions of the Future](https://mlsanity.com/paper/2604.16451/) (2026)
- [Earth Science Foundation Models: From Perception to Reasoning and Discovery](https://mlsanity.com/paper/2605.12542/) (2026)
- [When Does Text Inform? Benchmarking Information-Theoretic Metrics for Multimodal Time-Series Forecasting](https://mlsanity.com/paper/2609.11282/) (2026)
- [EarthAlign: Learning Language-Aligned Representations of Earth System Data](https://mlsanity.com/paper/2610.14624/) (2026)

## Abstract (arXiv)

Forecasting weather and climate events is crucial for making appropriate measures to mitigate environmental hazards and minimize losses. However, existing environmental forecasting research focuses narrowly on predicting numerical meteorological variables (e.g., temperature), neglecting the translation of these variables into actionable textual narratives of events and their consequences. To bridge this gap, we proposed Weather and Climate Event Forecasting (WCEF), a new task that leverages numerical meteorological raster data and textual event data to predict weather and climate events. This task is challenging to accomplish due to difficulties in aligning multimodal data and the lack of supervised datasets. To address these challenges, we present CLLMate, the first multimodal dataset for WCEF, using 26,156 environmental news articles aligned with ERA5 reanalysis data. We systematically benchmark 23 existing MLLMs on CLLMate, including closed-source, open-source, and our fine-tuned models. Our experiments reveal the advantages and limitations of existing MLLMs and the value of CLLMate for the training and benchmarking of the WCEF task.