# GENEVA: Benchmarking Generalizability for Event Argument Extraction with Hundreds of Event Types and Argument Roles

arXiv: https://arxiv.org/abs/2205.12505 · HTML: https://mlsanity.com/paper/2205.12505/

## Key facts

- Cited by 26 papers in the map (data through 14 Oct 2026).
- Builds on 17 of its 53 references that are in the map.
- First posted on arXiv 25 May 2022.

## Builds on (most cited first, 17 of 17)

- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [PaLM: Scaling Language Modeling with Pathways](https://mlsanity.com/paper/2204.02311/) (2022)
- [Exploiting Cloze Questions for Few Shot Text Classification and Natural Language Inference](https://mlsanity.com/paper/2001.07676/) (2020)
- [It's Not Just Size That Matters: Small Language Models Are Also Few-Shot Learners](https://mlsanity.com/paper/2009.07118/) (2020)
- [Entity, Relation, and Event Extraction with Contextualized Span Representations](https://mlsanity.com/paper/1909.03546/) (2019)
- [Structured Prediction as Translation between Augmented Natural Languages](https://mlsanity.com/paper/2101.05779/) (2021)
- [Event Extraction by Answering (Almost) Natural Questions](https://mlsanity.com/paper/2004.13625/) (2020)
- [Document-Level Event Argument Extraction by Conditional Generation](https://mlsanity.com/paper/2104.05919/) (2021)
- [MAVEN: A Massive General Domain Event Detection Dataset](https://mlsanity.com/paper/2004.13590/) (2020)
- [DEGREE: A Data-Efficient Generation-Based Event Extraction Model](https://mlsanity.com/paper/2108.12724/) (2021)
- [Multi-Sentence Argument Linking](https://mlsanity.com/paper/1911.03766/) (2019)
- [Zero-Shot Transfer Learning for Event Extraction](https://mlsanity.com/paper/1707.01066/) (2017)
- [Joint Extraction of Events and Entities within a Document Context](https://mlsanity.com/paper/1609.03632/) (2016)
- [ESTER: A Machine Reading Comprehension Dataset for Event Semantic Relation Reasoning](https://mlsanity.com/paper/2104.08350/) (2021)
- [Unsupervised Label-aware Event Trigger and Argument Classification](https://mlsanity.com/paper/2012.15243/) (2020)
- [Query and Extract: Refining Event Extraction as Type-oriented Binary Decoding](https://mlsanity.com/paper/2110.07476/) (2021)
- [Textual Entailment for Event Argument Extraction: Zero- and Few-Shot with Multi-Source Learning](https://mlsanity.com/paper/2205.01376/) (2022)

## Built on by (most cited first, 26 of 26)

- [LLM-Ensemble: Optimal Large Language Model Ensemble Method for E-commerce Product Attribute Value Extraction](https://mlsanity.com/paper/2403.00863/) (2024)
- [TextEE: Benchmark, Reevaluation, Reflections, and Future Challenges in Event Extraction](https://mlsanity.com/paper/2311.09562/) (2023)
- [ExtractGPT: Exploring the Potential of Large Language Models for Product Attribute Value Extraction](https://mlsanity.com/paper/2310.12537/) (2023)
- [Using LLMs for the Extraction and Normalization of Product Attribute Values](https://mlsanity.com/paper/2403.02130/) (2024)
- [Large Language Models in Argument Mining: A Survey](https://mlsanity.com/paper/2506.16383/) (2025)
- [Contextual Label Projection for Cross-Lingual Structured Prediction](https://mlsanity.com/paper/2309.08943/) (2023)
- [Improving Event Definition Following For Zero-Shot Event Detection](https://mlsanity.com/paper/2403.02586/) (2024)
- [Event Detection from Social Media for Epidemic Prediction](https://mlsanity.com/paper/2404.01679/) (2024)
- [SPEED++: A Multilingual Event Extraction Framework for Epidemic Prediction and Preparedness](https://mlsanity.com/paper/2410.18393/) (2024)
- [Adaptive Schema-aware Event Extraction with Retrieval-Augmented Generation](https://mlsanity.com/paper/2505.08690/) (2025)
- [MAVEN-Arg: Completing the Puzzle of All-in-One Event Understanding Dataset with Event Argument Annotation](https://mlsanity.com/paper/2311.09105/) (2023)
- [Event Extraction in Large Language Model](https://mlsanity.com/paper/2512.19537/) (2025)
- [REGen: A Reliable Evaluation Framework for Generative Event Argument Extraction](https://mlsanity.com/paper/2502.16838/) (2025)
- [Exploring Large Language Models for Product Attribute Value Identification](https://mlsanity.com/paper/2409.12695/) (2024)
- [SNaRe: Domain-aware Data Generation for Low-Resource Event Detection](https://mlsanity.com/paper/2502.17394/) (2025)
- [GenEARL: A Training-Free Generative Framework for Multimodal Event Argument Role Labeling](https://mlsanity.com/paper/2404.04763/) (2024)
- [Self-Refinement Strategies for LLM-based Product Attribute Value Extraction](https://mlsanity.com/paper/2501.01237/) (2025)
- [Argument-Aware Approach To Event Linking](https://mlsanity.com/paper/2403.15097/) (2024)
- [EXCEEDS: Extracting Complex Events via Nugget-based Grid Modeling in Scientific Domain](https://mlsanity.com/paper/2406.14075/) (2024)
- [Automated Archival Descriptions with Federated Intelligence of LLMs](https://mlsanity.com/paper/2504.05711/) (2025)
- [Improving Information Extraction with Learned Queries](https://mlsanity.com/paper/2608.31058/) (2026)
- [Large Language Models for Document-Level Event-Argument Data Augmentation for Challenging Role Types](https://mlsanity.com/paper/2403.03304/) (2024)
- [Beyond Exact Match: Semantically Reassessing Event Extraction by Large Language Models](https://mlsanity.com/paper/2410.09418/) (2024)
- [EVENT5Ws: A Large Dataset for Open-Domain Event Extraction from Documents](https://mlsanity.com/paper/2604.21890/) (2026)
- [A Multi-Domain and Multi-Task Generative Framework with Explicit Task and Domain Conditioning for Cross-Domain Event Extraction](https://mlsanity.com/paper/2608.23235/) (2026)
- [Calibrated Typed Decisions for Entity, Relation, and Event Extraction](https://mlsanity.com/paper/2610.17113/) (2026)

## Abstract (arXiv)

Recent works in Event Argument Extraction (EAE) have focused on improving model generalizability to cater to new events and domains. However, standard benchmarking datasets like ACE and ERE cover less than 40 event types and 25 entity-centric argument roles. Limited diversity and coverage hinder these datasets from adequately evaluating the generalizability of EAE models. In this paper, we first contribute by creating a large and diverse EAE ontology. This ontology is created by transforming FrameNet, a comprehensive semantic role labeling (SRL) dataset for EAE, by exploiting the similarity between these two tasks. Then, exhaustive human expert annotations are collected to build the ontology, concluding with 115 events and 220 argument roles, with a significant portion of roles not being entities. We utilize this ontology to further introduce GENEVA, a diverse generalizability benchmarking dataset comprising four test suites, aimed at evaluating models' ability to handle limited data and unseen event type generalization. We benchmark six EAE models from various families. The results show that owing to non-entity argument roles, even the best-performing model can only achieve 39% F1 score, indicating how GENEVA provides new challenges for generalization in EAE. Overall, our large and diverse EAE ontology can aid in creating more comprehensive future resources, while GENEVA is a challenging benchmarking dataset encouraging further research for improving generalizability in EAE. The code and data can be found at https://github.com/PlusLabNLP/GENEVA.