# Evaluating Robustness of LLMs on Crisis-Related Microblogs across Events, Information Types, and Linguistic Features

arXiv: https://arxiv.org/abs/2412.10413 · HTML: https://mlsanity.com/paper/2412.10413/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- Builds on 12 of its 37 references that are in the map.
- First posted on arXiv 8 Dec 2024.

## Builds on (most cited first, 12 of 12)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [The Llama 3 Herd of Models](https://mlsanity.com/paper/2407.21783/) (2024)
- [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://mlsanity.com/paper/2307.09288/) (2023)
- [DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter](https://mlsanity.com/paper/1910.01108/) (2019)
- [Processing Social Media Messages in Mass Emergency: A Survey](https://mlsanity.com/paper/1407.7071/) (2014)
- [Twitter as a Lifeline: Human-annotated Twitter Corpora for NLP of Crisis-related Messages](https://mlsanity.com/paper/1605.05894/) (2016)
- [HumAID: Human-Annotated Disaster Incidents Data from Twitter with Deep Learning Benchmarks](https://mlsanity.com/paper/2104.03090/) (2021)
- [LLM-Assisted Crisis Management: Building Advanced LLM Platforms for Effective Emergency Response and Public Collaboration](https://mlsanity.com/paper/2402.10908/) (2024)
- [A Twitter Tale of Three Hurricanes: Harvey, Irma, and Maria](https://mlsanity.com/paper/1805.05144/) (2018)
- [QuakeBERT: Accurate Classification of Social Media Texts for Rapid Earthquake Impact Assessment](https://mlsanity.com/paper/2405.06684/) (2024)
- [Monitoring Critical Infrastructure Facilities During Disasters Using Large Language Models](https://mlsanity.com/paper/2404.14432/) (2024)

## Built on by (most cited first, 7 of 7)

- [Detecting Actionable Requests and Offers on Social Media During Crises Using LLMs](https://mlsanity.com/paper/2504.16144/) (2025)
- [OnRL-RAG: Real-Time Personalized Mental Health Dialogue System](https://mlsanity.com/paper/2504.02894/) (2025)
- [LLM-Powered Automatic Translation and Urgency in Crisis Scenarios](https://mlsanity.com/paper/2602.13452/) (2026)
- [Extracting Disaster Impacts and Impact Related Locations in Social Media Posts Using Large Language Models](https://mlsanity.com/paper/2511.21753/) (2025)
- [Cross-Modal Rationale Transfer for Explainable Humanitarian Classification on Social Media](https://mlsanity.com/paper/2603.18611/) (2026)
- [LLM-guided Semi-Supervised Approaches for Social Media Crisis Data Classification](https://mlsanity.com/paper/2605.08448/) (2026)
- [CrisisFake: Benchmark Validity of AI-Generated Text Detection for Disaster Social Sensing](https://mlsanity.com/paper/2609.35821/) (2026)

## Abstract (arXiv)

The widespread use of microblogging platforms like X (formerly Twitter) during disasters provides real-time information to governments and response authorities. However, the data from these platforms is often noisy, requiring automated methods to filter relevant information. Traditionally, supervised machine learning models have been used, but they lack generalizability. In contrast, Large Language Models (LLMs) show better capabilities in understanding and processing natural language out of the box. This paper provides a detailed analysis of the performance of six well-known LLMs in processing disaster-related social media data from a large-set of real-world events. Our findings indicate that while LLMs, particularly GPT-4o and GPT-4, offer better generalizability across different disasters and information types, most LLMs face challenges in processing flood-related data, show minimal improvement despite the provision of examples (i.e., shots), and struggle to identify critical information categories like urgent requests and needs. Additionally, we examine how various linguistic features affect model performance and highlight LLMs' vulnerabilities against certain features like typos. Lastly, we provide benchmarking results for all events across both zero- and few-shot settings and observe that proprietary models outperform open-source ones in all tasks.