# A Multi-Axis Annotation Scheme for Event Temporal Relations

arXiv: https://arxiv.org/abs/1804.07828 · HTML: https://mlsanity.com/paper/1804.07828/

## Key facts

- Cited by 75 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 20 Apr 2018.

## Built on by (most cited first, 30 of 75)

- [Tailor: Generating and Perturbing Text with Semantic Controls](https://mlsanity.com/paper/2107.07150/) (2021)
- [MAVEN-ERE: A Unified Large-scale Dataset for Event Coreference, Temporal, Causal, and Subevent Relation Extraction](https://mlsanity.com/paper/2211.07342/) (2022)
- [Zero-shot Temporal Relation Extraction with ChatGPT](https://mlsanity.com/paper/2304.05454/) (2023)
- [Language Models Can Improve Event Prediction by Few-Shot Abductive Reasoning](https://mlsanity.com/paper/2305.16646/) (2023)
- [ChatGPT Evaluation on Sentence Level Relations: A Focus on Temporal, Causal, and Discourse Relations](https://mlsanity.com/paper/2304.14827/) (2023)
- [When does In-context Learning Fall Short and Why? A Study on Specification-Heavy Tasks](https://mlsanity.com/paper/2311.08993/) (2023)
- [ASER: Towards Large-scale Commonsense Knowledge Acquisition via Higher-order Selectional Preference over Eventualities](https://mlsanity.com/paper/2104.02137/) (2021)
- [ADELIE: Aligning Large Language Models on Information Extraction](https://mlsanity.com/paper/2405.05008/) (2024)
- [COLA: Contextualized Commonsense Causal Reasoning from the Causal Inference Perspective](https://mlsanity.com/paper/2305.05191/) (2023)
- [NewsEdits: A News Article Revision Dataset and a Document-Level Reasoning Challenge](https://mlsanity.com/paper/2206.07106/) (2022)
- [Generic Temporal Reasoning with Differential Analysis and Explanation](https://mlsanity.com/paper/2212.10467/) (2022)
- [BIRD: A Trustworthy Bayesian Inference Framework for Large Language Models](https://mlsanity.com/paper/2404.12494/) (2024)
- [EventPlus: A Temporal Event Understanding Pipeline](https://mlsanity.com/paper/2101.04922/) (2021)
- [Acquiring and Modelling Abstract Commonsense Knowledge via Conceptualization](https://mlsanity.com/paper/2206.01532/) (2022)
- [Extracting Temporal Event Relation with Syntax-guided Graph Transformer](https://mlsanity.com/paper/2104.09570/) (2021)
- [Go Back in Time: Generating Flashbacks in Stories with Event Temporal Prompts](https://mlsanity.com/paper/2205.01898/) (2022)
- [Extracting or Guessing? Improving Faithfulness of Event Temporal Relation Extraction](https://mlsanity.com/paper/2210.04992/) (2022)
- [More than Classification: A Unified Framework for Event Temporal Relation Extraction](https://mlsanity.com/paper/2305.17607/) (2023)
- [Extracting Event Temporal Relations via Hyperbolic Geometry](https://mlsanity.com/paper/2109.05527/) (2021)
- [Getting Sick After Seeing a Doctor? Diagnosing and Mitigating Knowledge Conflicts in Event Temporal Reasoning](https://mlsanity.com/paper/2305.14970/) (2023)
- [Improving Event Representation via Simultaneous Weakly Supervised Contrastive Learning and Clustering](https://mlsanity.com/paper/2203.07633/) (2022)
- [OmniEvent: A Comprehensive, Fair, and Easy-to-Use Toolkit for Event Understanding](https://mlsanity.com/paper/2309.14258/) (2023)
- [Once Upon a textitTime in textitGraph: Relative-Time Pretraining for Complex Temporal Reasoning](https://mlsanity.com/paper/2310.14709/) (2023)
- [Event Temporal Relation Extraction with Bayesian Translational Model](https://mlsanity.com/paper/2302.04985/) (2023)
- [An Overview Of Temporal Commonsense Reasoning and Acquisition](https://mlsanity.com/paper/2308.00002/) (2023)
- [TIMELINE: Exhaustive Annotation of Temporal Relations Supporting the Automatic Ordering of Events in News Articles](https://mlsanity.com/paper/2310.17802/) (2023)
- [Set-Aligning Framework for Auto-Regressive Event Temporal Graph Generation](https://mlsanity.com/paper/2404.01532/) (2024)
- [Are LLMs Good Annotators for Discourse-level Event Relation Extraction?](https://mlsanity.com/paper/2407.19568/) (2024)
- [Distilling Multi-Scale Knowledge for Event Temporal Relation Extraction](https://mlsanity.com/paper/2209.00568/) (2022)
- [MAVEN-Arg: Completing the Puzzle of All-in-One Event Understanding Dataset with Event Argument Annotation](https://mlsanity.com/paper/2311.09105/) (2023)

## Abstract (arXiv)

Existing temporal relation (TempRel) annotation schemes often have low inter-annotator agreements (IAA) even between experts, suggesting that the current annotation task needs a better definition. This paper proposes a new multi-axis modeling to better capture the temporal structure of events. In addition, we identify that event end-points are a major source of confusion in annotation, so we also propose to annotate TempRels based on start-points only. A pilot expert annotation using the proposed scheme shows significant improvement in IAA from the conventional 60's to 80's (Cohen's Kappa). This better-defined annotation scheme further enables the use of crowdsourcing to alleviate the labor intensity for each annotator. We hope that this work can foster more interesting studies towards event understanding.