# DisasterQA: A Benchmark for Assessing the performance of LLMs in Disaster Response

arXiv: https://arxiv.org/abs/2410.20707 · HTML: https://mlsanity.com/paper/2410.20707/

## Key facts

- Cited by 9 papers in the map (data through 14 Oct 2026).
- Builds on 11 of its 26 references that are in the map.
- First posted on arXiv 9 Oct 2024.

## Builds on (most cited first, 11 of 11)

- [Chain-of-Thought Prompting Elicits Reasoning in Large Language Models](https://mlsanity.com/paper/2201.11903/) (2022)
- [Self-Consistency Improves Chain of Thought Reasoning in Language Models](https://mlsanity.com/paper/2203.11171/) (2022)
- [A Comprehensive Overview of Large Language Models](https://mlsanity.com/paper/2307.06435/) (2023)
- [Large Language Models Understand and Can be Enhanced by Emotional Stimuli](https://mlsanity.com/paper/2307.11760/) (2023)
- [Guiding Large Language Models via Directional Stimulus Prompting](https://mlsanity.com/paper/2302.11520/) (2023)
- [Fairness-guided Few-shot Prompting for Large Language Models](https://mlsanity.com/paper/2303.13217/) (2023)
- [DisasterResponseGPT: Large Language Models for Accelerated Plan of Action Development in Disaster Response Scenarios](https://mlsanity.com/paper/2306.17271/) (2023)
- [CrisisSense-LLM: Instruction Fine-Tuned Large Language Model for Multi-label Social Media Text Classification in Disaster Informatics](https://mlsanity.com/paper/2406.15477/) (2024)
- [LLM-Assisted Crisis Management: Building Advanced LLM Platforms for Effective Emergency Response and Public Collaboration](https://mlsanity.com/paper/2402.10908/) (2024)
- [FloodBrain: Flood Disaster Reporting by Web-based Retrieval Augmented Generation with an LLM](https://mlsanity.com/paper/2311.02597/) (2023)
- [Drones4Good: Supporting Disaster Relief Through Remote Sensing and AI](https://mlsanity.com/paper/2308.05074/) (2023)

## Built on by (most cited first, 9 of 9)

- [Harnessing Large Language Models for Disaster Management: A Survey](https://mlsanity.com/paper/2501.06932/) (2025)
- [A Survey of Scaling in Large Language Model Reasoning](https://mlsanity.com/paper/2504.02181/) (2025)
- [Domain Specific Benchmarks for Evaluating Multimodal Large Language Models](https://mlsanity.com/paper/2506.12958/) (2025)
- [DisastQA: A Comprehensive Benchmark for Evaluating Question Answering in Disaster Management](https://mlsanity.com/paper/2601.03670/) (2026)
- [DisasterInsight: A Building-Centric Benchmark for Evaluating Vision--Language Models in Disaster Response](https://mlsanity.com/paper/2601.18493/) (2026)
- [DisasterBench: A Multimodal Benchmark for UAV-Based Disaster Response in Complex Environments](https://mlsanity.com/paper/2606.06217/) (2026)
- [RAPID: A Reproducible Multi-Agent Pipeline for Interpretable Disaster Damage Assessment from Satellite and Street-View Imagery](https://mlsanity.com/paper/2606.21819/) (2026)
- [A Dynamic Fusion Model for Consistent Crisis Response](https://mlsanity.com/paper/2509.01053/) (2025)
- [A Lightweight LLM Framework for Disaster Humanitarian Information Classification](https://mlsanity.com/paper/2602.12284/) (2026)

## Abstract (arXiv)

Disasters can result in the deaths of many, making quick response times vital. Large Language Models (LLMs) have emerged as valuable in the field. LLMs can be used to process vast amounts of textual information quickly providing situational context during a disaster. However, the question remains whether LLMs should be used for advice and decision making in a disaster. To evaluate the capabilities of LLMs in disaster response knowledge, we introduce a benchmark: DisasterQA created from six online sources. The benchmark covers a wide range of disaster response topics. We evaluated five LLMs each with four different prompting methods on our benchmark, measuring both accuracy and confidence levels through Logprobs. The results indicate that LLMs require improvement on disaster response knowledge. We hope that this benchmark pushes forth further development of LLMs in disaster response, ultimately enabling these models to work alongside. emergency managers in disasters.