# RESPONSE: Benchmarking the Ability of Language Models to Undertake Commonsense Reasoning in Crisis Situation

arXiv: https://arxiv.org/abs/2503.11348 · HTML: https://mlsanity.com/paper/2503.11348/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Placed in Reasoning benchmarks and evaluation, one of 17 directions in LLM reasoning.
- Builds on 21 of its 31 references that are in the map.
- First posted on arXiv 14 Mar 2025.

## Placed in

- LLM reasoning / [Reasoning benchmarks and evaluation](https://mlsanity.com/lr/reasoning-benchmarks-and-evaluation/)

## Builds on (most cited first, 21 of 21)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [BERTScore: Evaluating Text Generation with BERT](https://mlsanity.com/paper/1904.09675/) (2019)
- [On the Opportunities and Risks of Foundation Models](https://mlsanity.com/paper/2108.07258/) (2021)
- [Generative Agents: Interactive Simulacra of Human Behavior](https://mlsanity.com/paper/2304.03442/) (2023)
- [Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question Answering](https://mlsanity.com/paper/2209.09513/) (2022)
- [Large Language Models Encode Clinical Knowledge](https://mlsanity.com/paper/2212.13138/) (2022)
- [A Call for Clarity in Reporting BLEU Scores](https://mlsanity.com/paper/1804.08771/) (2018)
- [BLEURT: Learning Robust Metrics for Text Generation](https://mlsanity.com/paper/2004.04696/) (2020)
- [Whose Opinions Do Language Models Reflect?](https://mlsanity.com/paper/2303.17548/) (2023)
- [Out of One, Many: Using Language Models to Simulate Human Samples](https://mlsanity.com/paper/2209.06899/) (2022)
- [Capabilities of GPT-4 on Medical Challenge Problems](https://mlsanity.com/paper/2303.13375/) (2023)
- [News Summarization and Evaluation in the Era of GPT-3](https://mlsanity.com/paper/2209.12356/) (2022)
- [Explaining Answers with Entailment Trees](https://mlsanity.com/paper/2104.08661/) (2021)
- [Learning to Retrieve Reasoning Paths over Wikipedia Graph for Question Answering](https://mlsanity.com/paper/1911.10470/) (2019)
- [When to Make Exceptions: Exploring Language Models as Accounts of Human Moral Judgment](https://mlsanity.com/paper/2210.01478/) (2022)
- [An Information-theoretic Approach to Prompt Engineering Without Ground Truth Labels](https://mlsanity.com/paper/2203.11364/) (2022)
- [Dungeons and Dragons as a Dialog Challenge for Artificial Intelligence](https://mlsanity.com/paper/2210.07109/) (2022)
- [DisasterResponseGPT: Large Language Models for Accelerated Plan of Action Development in Disaster Response Scenarios](https://mlsanity.com/paper/2306.17271/) (2023)
- [Incidents1M: a large-scale dataset of images with natural disasters, damage, and incidents](https://mlsanity.com/paper/2201.04236/) (2022)
- [Detecting natural disasters, damage, and incidents in the wild](https://mlsanity.com/paper/2008.09188/) (2020)

## Built on by (most cited first, 4 of 4)

- [Towards Evaluating Proactive Risk Awareness of Multimodal Language Models](https://mlsanity.com/paper/2505.17455/) (2025)
- [PaSBench-Video: A Streaming Video Benchmark for Proactive Safety Warning](https://mlsanity.com/paper/2606.02443/) (2026)
- [DisasterBench: A Multimodal Benchmark for UAV-Based Disaster Response in Complex Environments](https://mlsanity.com/paper/2606.06217/) (2026)
- [From Sports to Safety: Benchmarking Proactive Risk Inference in MLLMs](https://mlsanity.com/paper/2608.05560/) (2026)

## Abstract (arXiv)

An interesting class of commonsense reasoning problems arises when people are faced with natural disasters. To investigate this topic, we present textsfRESPONSE, a human-curated dataset containing 1789 annotated instances featuring 6037 sets of questions designed to assess LLMs' commonsense reasoning in disaster situations across different time frames. The dataset includes problem descriptions, missing resources, time-sensitive solutions, and their justifications, with a subset validated by environmental engineers. Through both automatic metrics and human evaluation, we compare LLM-generated recommendations against human responses. Our findings show that even state-of-the-art models like GPT-4 achieve only 37% human-evaluated correctness for immediate response actions, highlighting significant room for improvement in LLMs' ability for commonsense reasoning in crises.