# Document-Level Relation Extraction with Adaptive Focal Loss and Knowledge Distillation

arXiv: https://arxiv.org/abs/2203.10900 · HTML: https://mlsanity.com/paper/2203.10900/

## Key facts

- Cited by 39 papers in the map (data through 14 Oct 2026).
- Placed in Knowledge distillation for LLMs, one of 12 directions in Knowledge distillation.
- Builds on 25 of its 33 references that are in the map.
- First posted on arXiv 21 Mar 2022.

## Placed in

- Knowledge distillation / [Knowledge distillation for LLMs](https://mlsanity.com/distill/knowledge-distillation-for-llms/)

## Builds on (most cited first, 25 of 25)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [Unsupervised Cross-lingual Representation Learning at Scale](https://mlsanity.com/paper/1911.02116/) (2019)
- [Focal Loss for Dense Object Detection](https://mlsanity.com/paper/1708.02002/) (2017)
- [Axial-DeepLab: Stand-Alone Axial-Attention for Panoptic Segmentation](https://mlsanity.com/paper/2003.07853/) (2020)
- [Matching the Blanks: Distributional Similarity for Relation Learning](https://mlsanity.com/paper/1906.03158/) (2019)
- [DocRED: A Large-Scale Document-Level Relation Extraction Dataset](https://mlsanity.com/paper/1906.06127/) (2019)
- [Graph Convolution over Pruned Dependency Trees Improves Relation Extraction](https://mlsanity.com/paper/1809.10185/) (2018)
- [Document-Level Relation Extraction with Adaptive Thresholding and Localized Context Pooling](https://mlsanity.com/paper/2010.11304/) (2020)
- [Cross-Sentence N-ary Relation Extraction with Graph LSTMs](https://mlsanity.com/paper/1708.03743/) (2017)
- [Reasoning with Latent Structure Refinement for Document-Level Relation Extraction](https://mlsanity.com/paper/2005.06312/) (2020)
- [Double Graph Based Reasoning for Document-level Relation Extraction](https://mlsanity.com/paper/2009.13752/) (2020)
- [Coreferential Reasoning Learning for Language Representation](https://mlsanity.com/paper/2004.06870/) (2020)
- [Document-level Relation Extraction as Semantic Segmentation](https://mlsanity.com/paper/2106.03618/) (2021)
- [Learning from Context or Names? An Empirical Study on Neural Relation Extraction](https://mlsanity.com/paper/2010.01923/) (2020)
- [Connecting the Dots: Document-level Neural Relation Extraction with Edge-oriented Graphs](https://mlsanity.com/paper/1909.00228/) (2019)
- [Entity Structure Within and Throughout: Modeling Mention Dependencies for Document-Level Relation Extraction](https://mlsanity.com/paper/2102.10249/) (2021)
- [Distant Supervision for Relation Extraction beyond the Sentence Boundary](https://mlsanity.com/paper/1609.04873/) (2016)
- [Reinforcement Learning for Relation Classification from Noisy Data](https://mlsanity.com/paper/1808.08013/) (2018)
- [Simultaneously Self-Attending to All Mentions for Full-Abstract Biological Relation Extraction](https://mlsanity.com/paper/1802.10569/) (2018)
- [Fine-tune Bert for DocRED with Two-step Process](https://mlsanity.com/paper/1909.11898/) (2019)
- [HIN: Hierarchical Inference Network for Document-Level Relation Extraction](https://mlsanity.com/paper/2003.12754/) (2020)
- [SIRE: Separate Intra- and Inter-sentential Reasoning for Document-level Relation Extraction](https://mlsanity.com/paper/2106.01709/) (2021)
- [DSGAN: Generative Adversarial Training for Distant Supervision Relation Extraction](https://mlsanity.com/paper/1805.09929/) (2018)

## Built on by (most cited first, 30 of 39)

- [Revisiting DocRED -- Addressing the False Negative Problem in Relation Extraction](https://mlsanity.com/paper/2205.12696/) (2022)
- [DREEAM: Guiding Attention with Evidence for Improving Document-Level Relation Extraction](https://mlsanity.com/paper/2302.08675/) (2023)
- [ERGO: Event Relational Graph Transformer for Document-level Event Causality Identification](https://mlsanity.com/paper/2204.07434/) (2022)
- [Information Screening whilst Exploiting! Multimodal Relation Extraction with Feature Denoising and Multimodal Topic Modeling](https://mlsanity.com/paper/2305.11719/) (2023)
- [A Dataset for Hyper-Relational Extraction and a Cube-Filling Approach](https://mlsanity.com/paper/2211.10018/) (2022)
- [Multi-hop Evidence Retrieval for Cross-document Relation Extraction](https://mlsanity.com/paper/2212.10786/) (2022)
- [Sentence-Level or Token-Level? A Comprehensive Study on Knowledge Distillation](https://mlsanity.com/paper/2404.14827/) (2024)
- [Can Transformers Capture Spatial Relations between Objects?](https://mlsanity.com/paper/2403.00729/) (2024)
- [A Unified Positive-Unlabeled Learning Framework for Document-Level Relation Extraction with Different Levels of Labeling](https://mlsanity.com/paper/2210.08709/) (2022)
- [RAPL: A Relation-Aware Prototype Learning Approach for Few-Shot Document-Level Relation Extraction](https://mlsanity.com/paper/2310.15743/) (2023)
- [LLM with Relation Classifier for Document-Level Relation Extraction](https://mlsanity.com/paper/2408.13889/) (2024)
- [Towards Better Document-level Relation Extraction via Iterative Inference](https://mlsanity.com/paper/2211.14470/) (2022)
- [Uncertainty Guided Label Denoising for Document-level Distant Relation Extraction](https://mlsanity.com/paper/2305.11029/) (2023)
- [HistRED: A Historical Document-Level Relation Extraction Dataset](https://mlsanity.com/paper/2307.04285/) (2023)
- [Towards Integration of Discriminability and Robustness for Document-Level Relation Extraction](https://mlsanity.com/paper/2304.00824/) (2023)
- [Rethinking Document-Level Relation Extraction: A Reality Check](https://mlsanity.com/paper/2306.08953/) (2023)
- [A Positive-Unlabeled Metric Learning Framework for Document-Level Relation Extraction with Incomplete Labeling](https://mlsanity.com/paper/2306.14806/) (2023)
- [A Comprehensive Survey of Document-level Relation Extraction (2016-2023)](https://mlsanity.com/paper/2309.16396/) (2023)
- [PromptRE: Weakly-Supervised Document-Level Relation Extraction via Prompting-Based Data Programming](https://mlsanity.com/paper/2310.09265/) (2023)
- [Improving Long Tailed Document-Level Relation Extraction via Easy Relation Augmentation and Contrastive Learning](https://mlsanity.com/paper/2205.10511/) (2022)
- [Towards Robust Low-Resource Fine-Tuning with Multi-View Compressed Representations](https://mlsanity.com/paper/2211.08794/) (2022)
- [A Survey of Document-Level Information Extraction](https://mlsanity.com/paper/2309.13249/) (2023)
- [PRiSM: Enhancing Low-Resource Document-Level Relation Extraction with Relation-Aware Score Calibration](https://mlsanity.com/paper/2309.13869/) (2023)
- [Anaphor Assisted Document-Level Relation Extraction](https://mlsanity.com/paper/2310.18604/) (2023)
- [Continual Contrastive Finetuning Improves Low-Resource Relation Extraction](https://mlsanity.com/paper/2212.10823/) (2022)
- [Class-Adaptive Self-Training for Relation Extraction with Incompletely Annotated Training Data](https://mlsanity.com/paper/2306.09697/) (2023)
- [Document-Level In-Context Few-Shot Relation Extraction via Pre-Trained Language Models](https://mlsanity.com/paper/2310.11085/) (2023)
- [GADePo: Graph-Assisted Declarative Pooling Transformers for Document-Level Relation Extraction](https://mlsanity.com/paper/2308.14423/) (2023)
- [GEGA: Graph Convolutional Networks and Evidence Retrieval Guided Attention for Enhanced Document-level Relation Extraction](https://mlsanity.com/paper/2407.21384/) (2024)
- [TPN: Transferable Proto-Learning Network towards Few-shot Document-Level Relation Extraction](https://mlsanity.com/paper/2410.00412/) (2024)

## Abstract (arXiv)

Document-level Relation Extraction (DocRE) is a more challenging task compared to its sentence-level counterpart. It aims to extract relations from multiple sentences at once. In this paper, we propose a semi-supervised framework for DocRE with three novel components. Firstly, we use an axial attention module for learning the interdependency among entity-pairs, which improves the performance on two-hop relations. Secondly, we propose an adaptive focal loss to tackle the class imbalance problem of DocRE. Lastly, we use knowledge distillation to overcome the differences between human annotated data and distantly supervised data. We conducted experiments on two DocRE datasets. Our model consistently outperforms strong baselines and its performance exceeds the previous SOTA by 1.36 F1 and 1.46 Ign_F1 score on the DocRED leaderboard. Our code and data will be released at https://github.com/tonytan48/KD-DocRE.