# Distilling Multi-Scale Knowledge for Event Temporal Relation Extraction

arXiv: https://arxiv.org/abs/2209.00568 · HTML: https://mlsanity.com/paper/2209.00568/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- Builds on 22 of its 38 references that are in the map.
- First posted on arXiv 1 Sep 2022.

## Builds on (most cited first, 22 of 22)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Distilling the Knowledge in a Neural Network](https://mlsanity.com/paper/1503.02531/) (2015)
- [DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter](https://mlsanity.com/paper/1910.01108/) (2019)
- [Longformer: The Long-Document Transformer](https://mlsanity.com/paper/2004.05150/) (2020)
- [Exploring Simple Siamese Representation Learning](https://mlsanity.com/paper/2011.10566/) (2020)
- [Fast Graph Representation Learning with PyTorch Geometric](https://mlsanity.com/paper/1903.02428/) (2019)
- [Modeling Relational Data with Graph Convolutional Networks](https://mlsanity.com/paper/1703.06103/) (2017)
- [Big Bird: Transformers for Longer Sequences](https://mlsanity.com/paper/2007.14062/) (2020)
- [Deep Mutual Learning](https://mlsanity.com/paper/1706.00384/) (2017)
- [Patient Knowledge Distillation for BERT Model Compression](https://mlsanity.com/paper/1908.09355/) (2019)
- [Relational Graph Attention Networks](https://mlsanity.com/paper/1904.05811/) (2019)
- [A Multi-Axis Annotation Scheme for Event Temporal Relations](https://mlsanity.com/paper/1804.07828/) (2018)
- [MixKD: Towards Efficient Distillation of Large-scale Language Models](https://mlsanity.com/paper/2011.00593/) (2020)
- [Contrastive Distillation on Intermediate Representations for Language Model Compression](https://mlsanity.com/paper/2009.14167/) (2020)
- [Joint Event and Temporal Relation Extraction with Shared Representations and Structured Prediction](https://mlsanity.com/paper/1909.05360/) (2019)
- [Joint Constrained Learning for Event-Event Relation Extraction](https://mlsanity.com/paper/2010.06727/) (2020)
- [Towards Cross-modality Medical Image Segmentation with Online Mutual Knowledge Distillation](https://mlsanity.com/paper/2010.01532/) (2020)
- [An Improved Neural Baseline for Temporal Relation Extraction](https://mlsanity.com/paper/1909.00429/) (2019)
- [LRC-BERT: Latent-representation Contrastive Knowledge Distillation for Natural Language Understanding](https://mlsanity.com/paper/2012.07335/) (2020)
- [A Structured Learning Approach to Temporal Relation Extraction](https://mlsanity.com/paper/1906.04943/) (2019)
- [Severing the Edge Between Before and After: Neural Architectures for Temporal Ordering of Events](https://mlsanity.com/paper/2004.04295/) (2020)

## Built on by (most cited first, 6 of 6)

- [Zero-shot Temporal Relation Extraction with ChatGPT](https://mlsanity.com/paper/2304.05454/) (2023)
- [GDLLM: A Global Distance-aware Modeling Approach Based on Large Language Models for Event Temporal Relation Extraction](https://mlsanity.com/paper/2508.20828/) (2025)
- [Transformer-Based Temporal Information Extraction and Application: A Review](https://mlsanity.com/paper/2504.07470/) (2025)
- [LANTERN: Scalable Distillation of Large Language Models for Job-Person Fit and Explanation](https://mlsanity.com/paper/2510.05490/) (2025)
- [R2-CoD: Understanding Text-Graph Complementarity in Relational Reasoning via Knowledge Co-Distillation](https://mlsanity.com/paper/2508.01475/) (2025)
- [WISTERIA: Weak Implicit Signal-based Temporal Relation Extraction with Attention](https://mlsanity.com/paper/2603.23319/) (2026)

## Abstract (arXiv)

Event Temporal Relation Extraction (ETRE) is paramount but challenging. Within a discourse, event pairs are situated at different distances or the so-called proximity bands. The temporal ordering communicated about event pairs where at more remote (i.e., ``long'') or less remote (i.e., ``short'') proximity bands are encoded differently. SOTA models have tended to perform well on events situated at either short or long proximity bands, but not both. Nonetheless, real-world, natural texts contain all types of temporal event-pairs. In this paper, we present MulCo: Distilling Multi-Scale Knowledge via Contrastive Learning, a knowledge co-distillation approach that shares knowledge across multiple event pair proximity bands to improve performance on all types of temporal datasets. Our experimental results show that MulCo successfully integrates linguistic cues pertaining to temporal reasoning across both short and long proximity bands and achieves new state-of-the-art results on several ETRE benchmark datasets.