# Semi-automatic Data Enhancement for Document-Level Relation Extraction with Distant Supervision from Large Language Models

arXiv: https://arxiv.org/abs/2311.07314 · HTML: https://mlsanity.com/paper/2311.07314/

## Key facts

- Cited by 9 papers in the map (data through 14 Oct 2026).
- Builds on 16 of its 22 references that are in the map.
- First posted on arXiv 13 Nov 2023.

## Builds on (most cited first, 16 of 16)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [ChatGPT Outperforms Crowd-Workers for Text-Annotation Tasks](https://mlsanity.com/paper/2303.15056/) (2023)
- [DocRED: A Large-Scale Document-Level Relation Extraction Dataset](https://mlsanity.com/paper/1906.06127/) (2019)
- [DeID-GPT: Zero-shot Medical Text De-Identification by GPT-4](https://mlsanity.com/paper/2303.11032/) (2023)
- [Document-Level Relation Extraction with Adaptive Thresholding and Localized Context Pooling](https://mlsanity.com/paper/2010.11304/) (2020)
- [Reasoning with Latent Structure Refinement for Document-Level Relation Extraction](https://mlsanity.com/paper/2005.06312/) (2020)
- [Thinking about GPT-3 In-Context Learning for Biomedical IE? Think Again](https://mlsanity.com/paper/2203.08410/) (2022)
- [Document-level Relation Extraction as Semantic Segmentation](https://mlsanity.com/paper/2106.03618/) (2021)
- [Revisiting DocRED -- Addressing the False Negative Problem in Relation Extraction](https://mlsanity.com/paper/2205.12696/) (2022)
- [Structured information extraction from complex scientific text with fine-tuned large language models](https://mlsanity.com/paper/2212.05238/) (2022)
- [Global-to-Local Neural Networks for Document-Level Relation Extraction](https://mlsanity.com/paper/2009.10359/) (2020)
- [DREEAM: Guiding Attention with Evidence for Improving Document-Level Relation Extraction](https://mlsanity.com/paper/2302.08675/) (2023)
- [How to Unleash the Power of Large Language Models for Few-shot Relation Extraction?](https://mlsanity.com/paper/2305.01555/) (2023)
- [Does Recommend-Revise Produce Reliable Annotations? An Analysis on Missing Instances in DocRED](https://mlsanity.com/paper/2204.07980/) (2022)

## Built on by (most cited first, 9 of 9)

- [Large Language Models for Generative Information Extraction: A Survey](https://mlsanity.com/paper/2312.17617/) (2023)
- [Research Trends for the Interplay between Large Language Models and Knowledge Graphs](https://mlsanity.com/paper/2406.08223/) (2024)
- [Beyond Single-Event Extraction: Towards Efficient Document-Level Multi-Event Argument Extraction](https://mlsanity.com/paper/2405.01884/) (2024)
- [LLM with Relation Classifier for Document-Level Relation Extraction](https://mlsanity.com/paper/2408.13889/) (2024)
- [Large Language Model for Extracting Complex Contract Information in Industrial Scenes](https://mlsanity.com/paper/2507.06539/) (2025)
- [Legal Document Summarization: Enhancing Judicial Efficiency through Automation Detection](https://mlsanity.com/paper/2507.18952/) (2025)
- [Relation as a Prior: A Novel Paradigm for LLM-based Document-level Relation Extraction](https://mlsanity.com/paper/2511.08143/) (2025)
- [DOREMI: Optimizing Long Tail Predictions in Document-Level Relation Extraction](https://mlsanity.com/paper/2601.11190/) (2026)
- [CONSISTRE: A Unified Consistency-Aware Framework for Document-Level Relation Extraction with Large Language Models](https://mlsanity.com/paper/2607.24312/) (2026)

## Abstract (arXiv)

Document-level Relation Extraction (DocRE), which aims to extract relations from a long context, is a critical challenge in achieving fine-grained structural comprehension and generating interpretable document representations. Inspired by recent advances in in-context learning capabilities emergent from large language models (LLMs), such as ChatGPT, we aim to design an automated annotation method for DocRE with minimum human effort. Unfortunately, vanilla in-context learning is infeasible for document-level relation extraction due to the plenty of predefined fine-grained relation types and the uncontrolled generations of LLMs. To tackle this issue, we propose a method integrating a large language model (LLM) and a natural language inference (NLI) module to generate relation triples, thereby augmenting document-level relation datasets. We demonstrate the effectiveness of our approach by introducing an enhanced dataset known as DocGNRE, which excels in re-annotating numerous long-tail relation types. We are confident that our method holds the potential for broader applications in domain-specific relation type definitions and offers tangible benefits in advancing generalized language semantic comprehension.