# Multimodal Event Detection: Current Approaches and Defining the New Playground through LLMs and VLMs

arXiv: https://arxiv.org/abs/2505.10836 · HTML: https://mlsanity.com/paper/2505.10836/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 13 of its 27 references that are in the map.
- First posted on arXiv 16 May 2025.

## Builds on (most cited first, 13 of 13)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models](https://mlsanity.com/paper/2301.12597/) (2023)
- [ConvNeXt V2: Co-designing and Scaling ConvNets with Masked Autoencoders](https://mlsanity.com/paper/2301.00808/) (2023)
- [Smarter, Better, Faster, Longer: A Modern Bidirectional Encoder for Fast, Memory Efficient, and Long Context Finetuning and Inference](https://mlsanity.com/paper/2412.13663/) (2024)
- [Entity, Relation, and Event Extraction with Contextualized Span Representations](https://mlsanity.com/paper/1909.03546/) (2019)
- [Cross-media Structured Common Space for Multimedia Event Extraction](https://mlsanity.com/paper/2005.02472/) (2020)
- [Instruction-Tuning LLMs for Event Extraction with Annotation Guidelines](https://mlsanity.com/paper/2502.16377/) (2025)
- [Instances Need More Care: Rewriting Prompts for Instances with LLMs in the Loop Yields Better Zero-Shot Performance](https://mlsanity.com/paper/2310.02107/) (2023)
- [SPEED++: A Multilingual Event Extraction Framework for Epidemic Prediction and Preparedness](https://mlsanity.com/paper/2410.18393/) (2024)
- [MAILEX: Email Event and Argument Extraction](https://mlsanity.com/paper/2305.13469/) (2023)
- [Revisiting Prompt Optimization with Large Reasoning Models-A Case Study on Event Extraction](https://mlsanity.com/paper/2504.07357/) (2025)
- [UniVSE: Robust Visual Semantic Embeddings via Structured Semantic Representations](https://mlsanity.com/paper/1904.05521/) (2019)

## Built on by (most cited first, 1 of 1)

- [Batch Prompting Suppresses Overthinking Reasoning Under Constraint: How Batch Prompting Suppresses Overthinking in Reasoning Models](https://mlsanity.com/paper/2511.04108/) (2025)

## Abstract (arXiv)

In this paper, we study the challenges of detecting events on social media, where traditional unimodal systems struggle due to the rapid and multimodal nature of data dissemination. We employ a range of models, including unimodal ModernBERT and ConvNeXt-V2, multimodal fusion techniques, and advanced generative models like GPT-4o, and LLaVA. Additionally, we also study the effect of providing multimodal generative models (such as GPT-4o) with a single modality to assess their efficacy. Our results indicate that while multimodal approaches notably outperform unimodal counterparts, generative approaches despite having a large number of parameters, lag behind supervised methods in precision. Furthermore, we also found that they lag behind instruction-tuned models because of their inability to generate event classes correctly. During our error analysis, we discovered that common social media issues such as leet speak, text elongation, etc. are effectively handled by generative approaches but are hard to tackle using supervised approaches.