RESPONSE: Benchmarking the Ability of Language Models to Undertake Commonsense Reasoning in Crisis Situation
On the map: LLM reasoning → Reasoning benchmarks and evaluation
Abstract (arXiv)
An interesting class of commonsense reasoning problems arises when people are faced with natural disasters. To investigate this topic, we present textsfRESPONSE, a human-curated dataset containing 1789 annotated instances featuring 6037 sets of questions designed to assess LLMs' commonsense reasoning in disaster situations across different time frames. The dataset includes problem descriptions, missing resources, time-sensitive solutions, and their justifications, with a subset validated by environmental engineers. Through both automatic metrics and human evaluation, we compare LLM-generated recommendations against human responses. Our findings show that even state-of-the-art models like GPT-4 achieve only 37% human-evaluated correctness for immediate response actions, highlighting significant room for improvement in LLMs' ability for commonsense reasoning in crises.
Key facts. Cited by 3 papers in the map (data through 7 Oct 2026). Placed in Reasoning benchmarks and evaluation, one of 17 directions in LLM reasoning. Builds on 21 of its 31 references that are in the map. First posted on arXiv 14 Mar 2025.
Builds on
21 of 31 references.
References · 21
- 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
- 2023GPT-4 Technical Reportcited by 18,132
- 2019BERTScore: Evaluating Text Generation with BERTcited by 5,462
- 2021On the Opportunities and Risks of Foundation Modelscited by 3,961
- 2023Generative Agents: Interactive Simulacra of Human Behaviorcited by 3,500
- 2022Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question Answeringcited by 1,881
- 2022Large Language Models Encode Clinical Knowledgecited by 1,807
- 2018A Call for Clarity in Reporting BLEU Scorescited by 1,790
- 2020BLEURT: Learning Robust Metrics for Text Generationcited by 1,196
- 2023Whose Opinions Do Language Models Reflect?cited by 759
- 2022Out of One, Many: Using Language Models to Simulate Human Samplescited by 737
- 2023Capabilities of GPT-4 on Medical Challenge Problemscited by 630
9 more
- 2022News Summarization and Evaluation in the Era of GPT-3cited by 324
- 2021Explaining Answers with Entailment Treescited by 148
- 2019Learning to Retrieve Reasoning Paths over Wikipedia Graph for Question Answeringcited by 147
- 2022When to Make Exceptions: Exploring Language Models as Accounts of Human Moral Judgmentcited by 93
- 2022An Information-theoretic Approach to Prompt Engineering Without Ground Truth Labelscited by 88
- 2022Dungeons and Dragons as a Dialog Challenge for Artificial Intelligencecited by 35
- 2023DisasterResponseGPT: Large Language Models for Accelerated Plan of Action Development in Disaster Response Scenarioscited by 33
- 2022Incidents1M: a large-scale dataset of images with natural disasters, damage, and incidentscited by 25
- 2020Detecting natural disasters, damage, and incidents in the wildcited by 23
Built on by
Citing papers per year: 2025: 1; 2026: 2.
3 papers.
Similar, not cited
Close in wording, no citation either way
- 2025Towards Quantifying Commonsense Reasoning with Mechanistic Insights
- 2024From Data to Commonsense Reasoning: The Use of Large Language Models for Explainable AI
- 2023An Overview Of Temporal Commonsense Reasoning and Acquisition
- 2025HellaSwag-Pro: A Large-Scale Bilingual Benchmark for Evaluating the Robustness of LLMs in Commonsense Reasoning
- 2025ReTraceQA: Evaluating Reasoning Traces of Small Language Models in Commonsense Question Answering
- 2025Com2: A Causal-Guided Benchmark for Exploring Complex Commonsense Reasoning in Large Language Models
- 2023CRoW: Benchmarking Commonsense Reasoning in Real-World Tasks
- 2022ROCK: Causal Inference Principles for Reasoning about Commonsense Causality