# REGen: A Reliable Evaluation Framework for Generative Event Argument Extraction

arXiv: https://arxiv.org/abs/2502.16838 · HTML: https://mlsanity.com/paper/2502.16838/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 23 of its 41 references that are in the map.
- First posted on arXiv 24 Feb 2025.

## Builds on (most cited first, 23 of 23)

- [The Llama 3 Herd of Models](https://mlsanity.com/paper/2407.21783/) (2024)
- [Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks](https://mlsanity.com/paper/1908.10084/) (2019)
- [Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena](https://mlsanity.com/paper/2306.05685/) (2023)
- [Self-Consistency Improves Chain of Thought Reasoning in Language Models](https://mlsanity.com/paper/2203.11171/) (2022)
- [SQuAD: 100,000+ Questions for Machine Comprehension of Text](https://mlsanity.com/paper/1606.05250/) (2016)
- [G-Eval: NLG Evaluation using GPT-4 with Better Human Alignment](https://mlsanity.com/paper/2303.16634/) (2023)
- [Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phone](https://mlsanity.com/paper/2404.14219/) (2024)
- [Mixtral of Experts](https://mlsanity.com/paper/2401.04088/) (2024)
- [A Survey on LLM-as-a-Judge](https://mlsanity.com/paper/2411.15594/) (2024)
- [Plan-and-Solve Prompting: Improving Zero-Shot Chain-of-Thought Reasoning by Large Language Models](https://mlsanity.com/paper/2305.04091/) (2023)
- [Event Extraction by Answering (Almost) Natural Questions](https://mlsanity.com/paper/2004.13625/) (2020)
- [Document-Level Event Argument Extraction by Conditional Generation](https://mlsanity.com/paper/2104.05919/) (2021)
- [DEGREE: A Data-Efficient Generation-Based Event Extraction Model](https://mlsanity.com/paper/2108.12724/) (2021)
- [Exploring the Feasibility of ChatGPT for Event Extraction](https://mlsanity.com/paper/2303.03836/) (2023)
- [GPTEval: A Survey on Assessments of ChatGPT and GPT-4](https://mlsanity.com/paper/2308.12488/) (2023)
- [PHEE: A Dataset for Pharmacovigilance Event Extraction from Text](https://mlsanity.com/paper/2210.12560/) (2022)
- [TextEE: Benchmark, Reevaluation, Reflections, and Future Challenges in Event Extraction](https://mlsanity.com/paper/2311.09562/) (2023)
- [GENEVA: Benchmarking Generalizability for Event Argument Extraction with Hundreds of Event Types and Argument Roles](https://mlsanity.com/paper/2205.12505/) (2022)
- [The Devil is in the Details: On the Pitfalls of Event Extraction Evaluation](https://mlsanity.com/paper/2306.06918/) (2023)
- [Event Extraction as Question Generation and Answering](https://mlsanity.com/paper/2307.05567/) (2023)
- [ULTRA: Unleash LLMs' Potential for Event Argument Extraction through Hierarchical Modeling and Pair-wise Self-Refinement](https://mlsanity.com/paper/2401.13218/) (2024)
- [Leveraging ChatGPT in Pharmacovigilance Event Extraction: An Empirical Study](https://mlsanity.com/paper/2402.15663/) (2024)
- [Explicit, Implicit, and Scattered: Revisiting Event Extraction to Capture Complex Arguments](https://mlsanity.com/paper/2410.03594/) (2024)

## Built on by (most cited first, 4 of 4)

- [Learning to Generate and Extract: A Multi-Agent Collaboration Framework For Zero-shot Document-level Event Arguments Extraction](https://mlsanity.com/paper/2603.02909/) (2026)
- [Improving Information Extraction with Learned Queries](https://mlsanity.com/paper/2608.31058/) (2026)
- [CliniCIRCA: A Modular LLM Framework for Constructing Longitudinal Mental Health Patient Journeys from Raw EHR Narratives](https://mlsanity.com/paper/2609.19585/) (2026)
- [ARGUS: Role-Aware Event Knowledge Graphs for U.S. Employment-Discrimination Complaints](https://mlsanity.com/paper/2609.30184/) (2026)

## Abstract (arXiv)

Event argument extraction identifies arguments for predefined event roles in text. Existing work evaluates this task with exact match (EM), where predicted arguments must align exactly with annotated spans. While suitable for span-based models, this approach falls short for large language models (LLMs), which often generate diverse yet semantically accurate arguments. EM severely underestimates performance by disregarding valid variations. Furthermore, EM evaluation fails to capture implicit arguments (unstated but inferable) and scattered arguments (distributed across a document). These limitations underscore the need for an evaluation framework that better captures models' actual performance. To bridge this gap, we introduce REGen, a Reliable Evaluation framework for Generative event argument extraction. REGen combines the strengths of exact, relaxed, and LLM-based matching to better align with human judgment. Experiments on six datasets show that REGen reveals an average performance gain of +23.93 F1 over EM, reflecting capabilities overlooked by prior evaluation. Human validation further confirms REGen's effectiveness, achieving 87.67% alignment with human assessments of argument correctness.