# Are LLMs Good Annotators for Discourse-level Event Relation Extraction?

arXiv: https://arxiv.org/abs/2407.19568 · HTML: https://mlsanity.com/paper/2407.19568/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- Builds on 22 of its 51 references that are in the map.
- First posted on arXiv 28 Jul 2024.

## Builds on (most cited first, 22 of 22)

- [The Curious Case of Neural Text Degeneration](https://mlsanity.com/paper/1904.09751/) (2019)
- [Least-to-Most Prompting Enables Complex Reasoning in Large Language Models](https://mlsanity.com/paper/2205.10625/) (2022)
- [ChatIE: Zero-Shot Information Extraction via Chatting with ChatGPT](https://mlsanity.com/paper/2302.10205/) (2023)
- [A Categorical Archive of ChatGPT Failures](https://mlsanity.com/paper/2302.03494/) (2023)
- [Does Synthetic Data Generation of LLMs Help Clinical Text Mining?](https://mlsanity.com/paper/2303.04360/) (2023)
- [Cutting Down on Prompts and Parameters: Simple Few-Shot Learning with Language Models](https://mlsanity.com/paper/2106.13353/) (2021)
- [Event2Mind: Commonsense Inference on Events, Intents, and Reactions](https://mlsanity.com/paper/1805.06939/) (2018)
- [Exploring the Feasibility of ChatGPT for Event Extraction](https://mlsanity.com/paper/2303.03836/) (2023)
- [A Multi-Axis Annotation Scheme for Event Temporal Relations](https://mlsanity.com/paper/1804.07828/) (2018)
- [Is ChatGPT a Good Causal Reasoner? A Comprehensive Evaluation](https://mlsanity.com/paper/2305.07375/) (2023)
- [MAVEN-ERE: A Unified Large-scale Dataset for Event Coreference, Temporal, Causal, and Subevent Relation Extraction](https://mlsanity.com/paper/2211.07342/) (2022)
- [How Far are We from Effective Context Modeling? An Exploratory Study on Semantic Parsing in Context](https://mlsanity.com/paper/2002.00652/) (2020)
- [Zero-shot Temporal Relation Extraction with ChatGPT](https://mlsanity.com/paper/2304.05454/) (2023)
- [TransOMCS: From Linguistic Graphs to Commonsense Knowledge](https://mlsanity.com/paper/2005.00206/) (2020)
- [VISAR: A Human-AI Argumentative Writing Assistant with Visual Programming and Rapid Draft Prototyping](https://mlsanity.com/paper/2304.07810/) (2023)
- [Coreference Resolution through a seq2seq Transition-Based System](https://mlsanity.com/paper/2211.12142/) (2022)
- [A Structured Learning Approach to Temporal Relation Extraction](https://mlsanity.com/paper/1906.04943/) (2019)
- [Question Answering as Global Reasoning over Semantic Abstractions](https://mlsanity.com/paper/1906.03672/) (2019)
- [Integrating Deep Event-Level and Script-Level Information for Script Event Prediction](https://mlsanity.com/paper/2110.15706/) (2021)
- [ProtoEM: A Prototype-Enhanced Matching Framework for Event Relation Extraction](https://mlsanity.com/paper/2309.12892/) (2023)
- [Guided scenarios with simulated expert personae: a remarkable strategy to perform cognitive work](https://mlsanity.com/paper/2306.03104/) (2023)
- [Leveraging Multiple Teachers for Test-Time Adaptation of Language-Guided Classifiers](https://mlsanity.com/paper/2311.07538/) (2023)

## Built on by (most cited first, 7 of 7)

- [Instruction-Tuning LLMs for Event Extraction with Annotation Guidelines](https://mlsanity.com/paper/2502.16377/) (2025)
- [GraphEval: A Lightweight Graph-Based LLM Framework for Idea Evaluation](https://mlsanity.com/paper/2503.12600/) (2025)
- [Token-Driven GammaTune: Adaptive Calibration for Enhanced Speculative Decoding](https://mlsanity.com/paper/2504.00030/) (2025)
- [LegalCore: A Dataset for Event Coreference Resolution in Legal Documents](https://mlsanity.com/paper/2502.12509/) (2025)
- [Is It JUST Semantics? A Case Study of Discourse Particle Understanding in LLMs](https://mlsanity.com/paper/2506.04534/) (2025)
- [Beyond Pairwise: Global Zero-shot Temporal Graph Generation](https://mlsanity.com/paper/2502.11114/) (2025)
- [Just Use XML: Revisiting Joint Translation and Label Projection](https://mlsanity.com/paper/2603.12021/) (2026)

## Abstract (arXiv)

Large Language Models (LLMs) have demonstrated proficiency in a wide array of natural language processing tasks. However, its effectiveness over discourse-level event relation extraction (ERE) tasks remains unexplored. In this paper, we assess the effectiveness of LLMs in addressing discourse-level ERE tasks characterized by lengthy documents and intricate relations encompassing coreference, temporal, causal, and subevent types. Evaluation is conducted using an commercial model, GPT-3.5, and an open-source model, LLaMA-2. Our study reveals a notable underperformance of LLMs compared to the baseline established through supervised learning. Although Supervised Fine-Tuning (SFT) can improve LLMs performance, it does not scale well compared to the smaller supervised baseline model. Our quantitative and qualitative analysis shows that LLMs have several weaknesses when applied for extracting event relations, including a tendency to fabricate event mentions, and failures to capture transitivity rules among relations, detect long distance relations, or comprehend contexts with dense event mentions. Code available at: https://github.com/WeiKangda/LLM-ERE.git.