# K-MetBench: A Multi-Dimensional Benchmark for Fine-Grained Evaluation of Expert Reasoning, Locality, and Multimodality in Meteorology

arXiv: https://arxiv.org/abs/2604.24645 · HTML: https://mlsanity.com/paper/2604.24645/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Placed in Time series reasoning, one of 17 directions in LLM reasoning.
- Builds on 15 of its 32 references that are in the map.
- First posted on arXiv 27 Apr 2026.

## Placed in

- LLM reasoning / [Time series reasoning](https://mlsanity.com/lr/time-series-reasoning/)

## Builds on (most cited first, 15 of 15)

- [Qwen3 Technical Report](https://mlsanity.com/paper/2505.09388/) (2025)
- [Qwen2.5-VL Technical Report](https://mlsanity.com/paper/2502.13923/) (2025)
- [InternVL3.5: Advancing Open-Source Multimodal Models in Versatility, Reasoning, and Efficiency](https://mlsanity.com/paper/2508.18265/) (2025)
- [gpt-oss-120b & gpt-oss-20b Model Card](https://mlsanity.com/paper/2508.10925/) (2025)
- [Show Your Work: Scratchpads for Intermediate Computation with Language Models](https://mlsanity.com/paper/2112.00114/) (2021)
- [Gemini: A Family of Highly Capable Multimodal Models](https://mlsanity.com/paper/2312.11805/) (2023)
- [Galactica: A Large Language Model for Science](https://mlsanity.com/paper/2211.09085/) (2022)
- [LLMs-as-Judges: A Comprehensive Survey on LLM-based Evaluation Methods](https://mlsanity.com/paper/2412.05579/) (2024)
- [Holistic Evaluation of Language Models](https://mlsanity.com/paper/2211.09110/) (2022)
- [Command A: An Enterprise-Ready Large Language Model](https://mlsanity.com/paper/2504.00698/) (2025)
- [EXAONE 4.0: Unified Large Language Models Integrating Non-reasoning and Reasoning Modes](https://mlsanity.com/paper/2507.11407/) (2025)
- [WeatherQA: Can Multimodal Language Models Reason about Severe Weather?](https://mlsanity.com/paper/2406.11217/) (2024)
- [ClimaQA: An Automated Evaluation Framework for Climate Question Answering Models](https://mlsanity.com/paper/2410.16701/) (2024)
- [HyperCLOVA X Technical Report](https://mlsanity.com/paper/2404.01954/) (2024)
- [VARCO-VISION-2.0 Technical Report](https://mlsanity.com/paper/2509.10105/) (2025)

## Abstract (arXiv)

The development of practical (multimodal) large language model assistants for Korean weather forecasters is hindered by the absence of a multidimensional, expert-level evaluation framework grounded in authoritative sources. To address this, we introduce K-MetBench, a diagnostic benchmark grounded in national qualification exams. It exposes critical gaps across four dimensions: expert visual reasoning of charts, logical validity via expert-verified rationales, Korean-specific geo-cultural comprehension, and fine-grained domain analysis. Our evaluation of 55 models reveals a profound modality gap in interpreting specialized diagrams and a reasoning gap where models hallucinate logic despite correct predictions. Crucially, Korean models outperform significantly larger global models in local contexts, demonstrating that parameter scaling alone cannot resolve cultural dependencies. K-MetBench serves as a roadmap for developing reliable, culturally aware expert AI agents. The dataset is available at https://huggingface.co/datasets/soyeonbot/K-MetBench .