# Evaluation Pitfalls and Challenges in Multimedia Event Extraction

arXiv: https://arxiv.org/abs/2606.26775 · HTML: https://mlsanity.com/paper/2606.26775/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 26 of its 48 references that are in the map.
- First posted on arXiv 25 Jun 2026.

## Builds on (most cited first, 26 of 26)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks](https://mlsanity.com/paper/1506.01497/) (2015)
- [You Only Look Once: Unified, Real-Time Object Detection](https://mlsanity.com/paper/1506.02640/) (2015)
- [Entity, Relation, and Event Extraction with Contextualized Span Representations](https://mlsanity.com/paper/1909.03546/) (2019)
- [MAVEN: A Massive General Domain Event Detection Dataset](https://mlsanity.com/paper/2004.13590/) (2020)
- [CLIP-Event: Connecting Text and Images with Event Structures](https://mlsanity.com/paper/2201.05078/) (2022)
- [Grounded Situation Recognition](https://mlsanity.com/paper/2003.12058/) (2020)
- [Cross-media Structured Common Space for Multimedia Event Extraction](https://mlsanity.com/paper/2005.02472/) (2020)
- [Visual Semantic Role Labeling for Video Understanding](https://mlsanity.com/paper/2104.00990/) (2021)
- [TextEE: Benchmark, Reevaluation, Reflections, and Future Challenges in Event Extraction](https://mlsanity.com/paper/2311.09562/) (2023)
- [Joint Multimedia Event Extraction from Video and Article](https://mlsanity.com/paper/2109.12776/) (2021)
- [The Devil is in the Details: On the Pitfalls of Event Extraction Evaluation](https://mlsanity.com/paper/2306.06918/) (2023)
- [UMIE: Unified Multimodal Information Extraction with Instruction Tuning](https://mlsanity.com/paper/2401.03082/) (2024)
- [Training Multimedia Event Extraction With Generated Images and Captions](https://mlsanity.com/paper/2306.08966/) (2023)
- [OmniEvent: A Comprehensive, Fair, and Easy-to-Use Toolkit for Event Understanding](https://mlsanity.com/paper/2309.14258/) (2023)
- [Grounding Partially-Defined Events in Multimodal Data](https://mlsanity.com/paper/2410.05267/) (2024)
- [Collaborative Multi-LoRA Experts with Achievement-based Multi-Tasks Loss for Unified Multimodal Information Extraction](https://mlsanity.com/paper/2505.06303/) (2025)
- [Multimodal Question Answering for Unified Information Extraction](https://mlsanity.com/paper/2310.03017/) (2023)
- [Recognizing Everything from All Modalities at Once: Grounded Multimodal Universal Information Extraction](https://mlsanity.com/paper/2406.03701/) (2024)
- [MMUTF: Multimodal Multimedia Event Argument Extraction with Unified Template Filling](https://mlsanity.com/paper/2406.12420/) (2024)
- [Multimodal Cross-Document Event Coreference Resolution Using Linear Semantic Transfer and Mixed-Modality Ensembles](https://mlsanity.com/paper/2404.08949/) (2024)
- [SpeechEE: A Novel Benchmark for Speech Event Extraction](https://mlsanity.com/paper/2408.09462/) (2024)
- [Benchmarking and Improving LVLMs on Event Extraction from Multimedia Documents](https://mlsanity.com/paper/2509.12876/) (2025)
- [Stepwise Schema-Guided Prompting Framework with Parameter Efficient Instruction Tuning for Multimedia Event Extraction](https://mlsanity.com/paper/2512.02584/) (2025)

## Abstract (arXiv)

Multimedia event extraction aims to jointly identify events and their arguments across multiple modalities, such as text and images, to support more comprehensive event understanding. While recent work reports steady and substantial progress, the reliability and comparability of these results critically depend on consistent and rigorous evaluation. In this work, we present the first systematic analysis of evaluation pitfalls in multimedia event extraction and identify three major sources of issues: inconsistent data processing, inconsistent task assumptions, and overly relaxed evaluation settings. We demonstrate, through a series of controlled experiments under a strict evaluation framework, that minor evaluation choices can cause large performance variations and lead to overestimation of a model's ability to ground real-world events across modalities. Our findings highlight the need for comparable evaluation standards and encourage a shift toward more rigorous evaluation in multimedia event extraction.