mlsanityData to 14 Oct 2026

A Multi-Axis Annotation Scheme for Event Temporal Relations

arXiv 1804.07828 ↗First posted 20 Apr 2018cs.CLcited by 75

Not on the map. Why?

Abstract (arXiv)

Existing temporal relation (TempRel) annotation schemes often have low inter-annotator agreements (IAA) even between experts, suggesting that the current annotation task needs a better definition. This paper proposes a new multi-axis modeling to better capture the temporal structure of events. In addition, we identify that event end-points are a major source of confusion in annotation, so we also propose to annotate TempRels based on start-points only. A pilot expert annotation using the proposed scheme shows significant improvement in IAA from the conventional 60's to 80's (Cohen's Kappa). This better-defined annotation scheme further enables the use of crowdsourcing to alleviate the labor intensity for each annotator. We hope that this work can foster more interesting studies towards event understanding.

Key facts. Cited by 75 papers in the map (data through 14 Oct 2026). First posted on arXiv 20 Apr 2018.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2021: 8; 2022: 15; 2023: 20; 2024: 13; 2025: 11; 2026: 8.

75 papers.

Sort:

By area:

  1. 2021Tailor: Generating and Perturbing Text with Semantic Controlscited by 68
  2. 2022MAVEN-ERE: A Unified Large-scale Dataset for Event Coreference, Temporal, Causal, and Subevent Relation Extractioncited by 66
  3. 2023Zero-shot Temporal Relation Extraction with ChatGPTcited by 53
  4. 2023Language Models Can Improve Event Prediction by Few-Shot Abductive Reasoningcited by 45
  5. 2023ChatGPT Evaluation on Sentence Level Relations: A Focus on Temporal, Causal, and Discourse Relationscited by 40
  6. 2023When does In-context Learning Fall Short and Why? A Study on Specification-Heavy Taskscited by 37
  7. 2021ASER: Towards Large-scale Commonsense Knowledge Acquisition via Higher-order Selectional Preference over Eventualitiescited by 28
  8. 2024ADELIE: Aligning Large Language Models on Information Extractioncited by 24
  9. 2023COLA: Contextualized Commonsense Causal Reasoning from the Causal Inference Perspectivecited by 22
  10. 2022NewsEdits: A News Article Revision Dataset and a Document-Level Reasoning Challengecited by 20
  11. 2022Generic Temporal Reasoning with Differential Analysis and Explanationcited by 18
  12. 2024BIRD: A Trustworthy Bayesian Inference Framework for Large Language Modelscited by 18
38 more
  1. 2021EventPlus: A Temporal Event Understanding Pipelinecited by 17
  2. 2022Acquiring and Modelling Abstract Commonsense Knowledge via Conceptualizationcited by 17
  3. 2021Extracting Temporal Event Relation with Syntax-guided Graph Transformercited by 16
  4. 2022Go Back in Time: Generating Flashbacks in Stories with Event Temporal Promptscited by 15
  5. 2022Extracting or Guessing? Improving Faithfulness of Event Temporal Relation Extractioncited by 15
  6. 2023More than Classification: A Unified Framework for Event Temporal Relation Extractioncited by 15
  7. 2021Extracting Event Temporal Relations via Hyperbolic Geometrycited by 13
  8. 2023Getting Sick After Seeing a Doctor? Diagnosing and Mitigating Knowledge Conflicts in Event Temporal Reasoningcited by 13
  9. 2022Improving Event Representation via Simultaneous Weakly Supervised Contrastive Learning and Clusteringcited by 11
  10. 2023OmniEvent: A Comprehensive, Fair, and Easy-to-Use Toolkit for Event Understandingcited by 11
  11. 2023Once Upon a textitTime in textitGraph: Relative-Time Pretraining for Complex Temporal Reasoningcited by 11
  12. 2023Event Temporal Relation Extraction with Bayesian Translational Modelcited by 10
  13. 2023An Overview Of Temporal Commonsense Reasoning and Acquisitioncited by 10
  14. 2023TIMELINE: Exhaustive Annotation of Temporal Relations Supporting the Automatic Ordering of Events in News Articlescited by 8
  15. 2024Set-Aligning Framework for Auto-Regressive Event Temporal Graph Generationcited by 8
  16. 2024Are LLMs Good Annotators for Discourse-level Event Relation Extraction?cited by 7
  17. 2022Distilling Multi-Scale Knowledge for Event Temporal Relation Extractioncited by 6
  18. 2023MAVEN-Arg: Completing the Puzzle of All-in-One Event Understanding Dataset with Event Argument Annotationcited by 6
  19. 2024Will LLMs Replace the Encoder-Only Models in Temporal Relation Classification?cited by 6
  20. 2022Event-Centric Question Answering via Contrastive Learning and Invertible Event Transformationcited by 5
  21. 2023tieval: An Evaluation Framework for Temporal Information Extraction Systemscited by 5
  22. 2025Event Extraction in Large Language Modelcited by 5
  23. 2022Learning Action Conditions from Instructional Manuals for Instruction Understandingcited by 4
  24. 2024Unveiling Divergent Inductive Biases of LLMs on Temporal Datacited by 4
  25. 2022Event Extraction: A Surveycited by 3
  26. 2023On Regularization and Inference with Label Constraintscited by 3
  27. 2023Fusing Temporal Graphs into Transformers for Time-Sensitive Question Answeringcited by 3
  28. 2024Sentence-level Media Bias Analysis with Event Relation Graphcited by 3
  29. 2025Beyond Pairwise: Global Zero-shot Temporal Graph Generationcited by 3
  30. 2025Chronocept: Instilling a Sense of Time in Machinescited by 3
  31. 2021Multitask Learning for Class-Imbalanced Discourse Classificationcited by 2
  32. 2021NewsEdits: A Dataset of Revision Histories for News Articles (Technical Report: Data Processing)cited by 2
  33. 2022NECE: Narrative Event Chain Extraction Toolkitcited by 2
  34. 2022Modeling Document-level Temporal Structures for Building Temporal Dependency Graphscited by 2
  35. 2023All Things Considered: Detecting Partisan Events from News Media with Cross-Article Comparisoncited by 2
  36. 2024Formulation Comparison for Timeline Construction using LLMscited by 2
  37. 2024EventFull: Complete and Consistent Event Relation Annotationcited by 2
  38. 2025MAQInstruct: Instruction-based Unified Event Relation Extractioncited by 2

50 most cited of 75 All 75 (JSON)