# The Devil is in the Details: On the Pitfalls of Event Extraction Evaluation

arXiv: https://arxiv.org/abs/2306.06918 · HTML: https://mlsanity.com/paper/2306.06918/

## Key facts

- Cited by 19 papers in the map (data through 14 Oct 2026).
- Builds on 51 of its 142 references that are in the map.
- First posted on arXiv 12 Jun 2023.

## Builds on (most cited first, 30 of 51)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension](https://mlsanity.com/paper/1910.13461/) (2019)
- [Entity, Relation, and Event Extraction with Contextualized Span Representations](https://mlsanity.com/paper/1909.03546/) (2019)
- [Event Extraction by Answering (Almost) Natural Questions](https://mlsanity.com/paper/2004.13625/) (2020)
- [Document-Level Event Argument Extraction by Conditional Generation](https://mlsanity.com/paper/2104.05919/) (2021)
- [Text2Event: Controllable Sequence-to-Structure Generation for End-to-end Event Extraction](https://mlsanity.com/paper/2106.09232/) (2021)
- [MAVEN: A Massive General Domain Event Detection Dataset](https://mlsanity.com/paper/2004.13590/) (2020)
- [ASER: A Large-scale Eventuality Knowledge Graph](https://mlsanity.com/paper/1905.00270/) (2019)
- [DEGREE: A Data-Efficient Generation-Based Event Extraction Model](https://mlsanity.com/paper/2108.12724/) (2021)
- [Multi-Sentence Argument Linking](https://mlsanity.com/paper/1911.03766/) (2019)
- [Jointly Multiple Events Extraction via Attention-based Graph Information Aggregation](https://mlsanity.com/paper/1809.09078/) (2018)
- [Doc2EDAG: An End-to-End Document-level Framework for Chinese Financial Event Extraction](https://mlsanity.com/paper/1904.07535/) (2019)
- [Prompt for Extraction? PAIE: Prompting Argument Interaction for Event Argument Extraction](https://mlsanity.com/paper/2202.12109/) (2022)
- [Joint Extraction of Events and Entities within a Document Context](https://mlsanity.com/paper/1609.03632/) (2016)
- [Multilingual Generative Language Models for Zero-Shot Cross-Lingual Event Argument Extraction](https://mlsanity.com/paper/2203.08308/) (2022)
- [CLEVE: Contrastive Pre-training for Event Extraction](https://mlsanity.com/paper/2105.14485/) (2021)
- [LEVEN: A Large-Scale Chinese Legal Event Detection Dataset](https://mlsanity.com/paper/2203.08556/) (2022)
- [Dynamic Prefix-Tuning for Generative Template-based Event Extraction](https://mlsanity.com/paper/2205.06166/) (2022)
- [Document-level Event Extraction via Heterogeneous Graph-based Interaction Model with a Tracker](https://mlsanity.com/paper/2105.14924/) (2021)
- [Document-Level Event Role Filler Extraction using Multi-Granularity Contextualized Encoding](https://mlsanity.com/paper/2005.06579/) (2020)
- [Unsupervised Label-aware Event Trigger and Argument Classification](https://mlsanity.com/paper/2012.15243/) (2020)
- [Query and Extract: Refining Event Extraction as Type-oriented Binary Decoding](https://mlsanity.com/paper/2110.07476/) (2021)
- [Textual Entailment for Event Argument Extraction: Zero- and Few-Shot with Multi-Source Learning](https://mlsanity.com/paper/2205.01376/) (2022)
- [Event Detection: Gate Diversity and Syntactic Importance Scoresfor Graph Convolution Neural Networks](https://mlsanity.com/paper/2010.14123/) (2020)
- [OntoED: Low-resource Event Detection with Ontology Embedding](https://mlsanity.com/paper/2105.10922/) (2021)
- [Incremental Prompting: Episodic Memory Prompt for Lifelong Event Detection](https://mlsanity.com/paper/2204.07275/) (2022)
- [Bidirectional Recurrent Neural Networks for Medical Event Detection in Electronic Health Records](https://mlsanity.com/paper/1606.07953/) (2016)
- [Few-Shot Event Detection with Prototypical Amortized Conditional Random Field](https://mlsanity.com/paper/2012.02353/) (2020)

## Built on by (most cited first, 19 of 19)

- [TextEE: Benchmark, Reevaluation, Reflections, and Future Challenges in Event Extraction](https://mlsanity.com/paper/2311.09562/) (2023)
- [ADELIE: Aligning Large Language Models on Information Extraction](https://mlsanity.com/paper/2405.05008/) (2024)
- [Simulating Public Administration Crisis: A Novel Generative Agent-Based Simulation System to Lower Technology Barriers in Social Science Research](https://mlsanity.com/paper/2311.06957/) (2023)
- [Mirror: A Universal Framework for Various Information Extraction Tasks](https://mlsanity.com/paper/2311.05419/) (2023)
- [OmniEvent: A Comprehensive, Fair, and Easy-to-Use Toolkit for Event Understanding](https://mlsanity.com/paper/2309.14258/) (2023)
- [Explicit, Implicit, and Scattered: Revisiting Event Extraction to Capture Complex Arguments](https://mlsanity.com/paper/2410.03594/) (2024)
- [MAVEN-Arg: Completing the Puzzle of All-in-One Event Understanding Dataset with Event Argument Annotation](https://mlsanity.com/paper/2311.09105/) (2023)
- [MAVEN-Fact: A Large-scale Event Factuality Detection Dataset](https://mlsanity.com/paper/2407.15352/) (2024)
- [REGen: A Reliable Evaluation Framework for Generative Event Argument Extraction](https://mlsanity.com/paper/2502.16838/) (2025)
- [Harvesting Events from Multiple Sources: Towards a Cross-Document Event Extraction Paradigm](https://mlsanity.com/paper/2406.16021/) (2024)
- [Mastering the Task of Open Information Extraction with Large Language Models and Consistent Reasoning Environment](https://mlsanity.com/paper/2310.10590/) (2023)
- [EXCEEDS: Extracting Complex Events via Nugget-based Grid Modeling in Scientific Domain](https://mlsanity.com/paper/2406.14075/) (2024)
- [Detecting Machine-Generated Long-Form Content with Latent-Space Variables](https://mlsanity.com/paper/2410.03856/) (2024)
- [Large Language Models for Document-Level Event-Argument Data Augmentation for Challenging Role Types](https://mlsanity.com/paper/2403.03304/) (2024)
- [Beyond Exact Match: Semantically Reassessing Event Extraction by Large Language Models](https://mlsanity.com/paper/2410.09418/) (2024)
- [NEST: Narrative Event Structures in Time for Long Video Understanding](https://mlsanity.com/paper/2606.19706/) (2026)
- [Evaluation Pitfalls and Challenges in Multimedia Event Extraction](https://mlsanity.com/paper/2606.26775/) (2026)
- [CMNIE: An Information Extraction Benchmark for Chinese Military News](https://mlsanity.com/paper/2609.10722/) (2026)
- [CLUE: A Contrastive Learning Framework for Unsupervised Event Type Induction and Annotation](https://mlsanity.com/paper/2610.12580/) (2026)

## Abstract (arXiv)

Event extraction (EE) is a crucial task aiming at extracting events from texts, which includes two subtasks: event detection (ED) and event argument extraction (EAE). In this paper, we check the reliability of EE evaluations and identify three major pitfalls: (1) The data preprocessing discrepancy makes the evaluation results on the same dataset not directly comparable, but the data preprocessing details are not widely noted and specified in papers. (2) The output space discrepancy of different model paradigms makes different-paradigm EE models lack grounds for comparison and also leads to unclear mapping issues between predictions and annotations. (3) The absence of pipeline evaluation of many EAE-only works makes them hard to be directly compared with EE works and may not well reflect the model performance in real-world pipeline scenarios. We demonstrate the significant influence of these pitfalls through comprehensive meta-analyses of recent papers and empirical experiments. To avoid these pitfalls, we suggest a series of remedies, including specifying data preprocessing, standardizing outputs, and providing pipeline evaluation results. To help implement these remedies, we develop a consistent evaluation framework OMNIEVENT, which can be obtained from https://github.com/THU-KEG/OmniEvent.