# Revisiting DocRED -- Addressing the False Negative Problem in Relation Extraction

arXiv: https://arxiv.org/abs/2205.12696 · HTML: https://mlsanity.com/paper/2205.12696/

## Key facts

- Cited by 57 papers in the map (data through 14 Oct 2026).
- Builds on 32 of its 54 references that are in the map.
- First posted on arXiv 25 May 2022.

## Builds on (most cited first, 30 of 32)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [Unsupervised Cross-lingual Representation Learning at Scale](https://mlsanity.com/paper/1911.02116/) (2019)
- [Symbolic Knowledge Distillation: from General Language Models to Commonsense Models](https://mlsanity.com/paper/2110.07178/) (2021)
- [FewRel: A Large-Scale Supervised Few-Shot Relation Classification Dataset with State-of-the-Art Evaluation](https://mlsanity.com/paper/1810.10147/) (2018)
- [DocRED: A Large-Scale Document-Level Relation Extraction Dataset](https://mlsanity.com/paper/1906.06127/) (2019)
- [Generating Datasets with Pretrained Language Models](https://mlsanity.com/paper/2104.07540/) (2021)
- [Not Enough Data? Deep Learning to the Rescue!](https://mlsanity.com/paper/1911.03118/) (2019)
- [Data Augmentation using Pre-trained Transformer Models](https://mlsanity.com/paper/2003.02245/) (2020)
- [Document-Level Relation Extraction with Adaptive Thresholding and Localized Context Pooling](https://mlsanity.com/paper/2010.11304/) (2020)
- [Reasoning with Latent Structure Refinement for Document-Level Relation Extraction](https://mlsanity.com/paper/2005.06312/) (2020)
- [TACRED Revisited: A Thorough Evaluation of the TACRED Relation Extraction Task](https://mlsanity.com/paper/2004.14855/) (2020)
- [FewRel 2.0: Towards More Challenging Few-Shot Relation Classification](https://mlsanity.com/paper/1910.07124/) (2019)
- [An Improved Baseline for Sentence-level Relation Extraction](https://mlsanity.com/paper/2102.01373/) (2021)
- [Document-level Relation Extraction as Semantic Segmentation](https://mlsanity.com/paper/2106.03618/) (2021)
- [Re-TACRED: Addressing Shortcomings of the TACRED Dataset](https://mlsanity.com/paper/2104.08398/) (2021)
- [Effective Modeling of Encoder-Decoder Architecture for Joint Entity and Relation Extraction](https://mlsanity.com/paper/1911.09886/) (2019)
- [SemEval-2010 Task 8: Multi-Way Classification of Semantic Relations Between Pairs of Nominals](https://mlsanity.com/paper/1911.10422/) (2019)
- [RelationPrompt: Leveraging Prompts to Generate Synthetic Data for Zero-Shot Relation Triplet Extraction](https://mlsanity.com/paper/2203.09101/) (2022)
- [More Data, More Relations, More Context and More Openness: A Review and Outlook for Relation Extraction](https://mlsanity.com/paper/2004.03186/) (2020)
- [MELM: Data Augmentation with Masked Entity Language Modeling for Low-Resource NER](https://mlsanity.com/paper/2108.13655/) (2021)
- [Document-Level Relation Extraction with Adaptive Focal Loss and Knowledge Distillation](https://mlsanity.com/paper/2203.10900/) (2022)
- [Empirical Analysis of Unlabeled Entity Problem in Named Entity Recognition](https://mlsanity.com/paper/2012.05426/) (2020)
- [An End-to-end Model for Entity-level Relation Extraction using Multi-instance Learning](https://mlsanity.com/paper/2102.05980/) (2021)
- [Tackling Long-Tailed Relations and Uncommon Entities in Knowledge Graph Completion](https://mlsanity.com/paper/1909.11359/) (2019)
- [Denoising Relation Extraction from Document-level Distant Supervision](https://mlsanity.com/paper/2011.03888/) (2020)
- [Does Recommend-Revise Produce Reliable Annotations? An Analysis on Missing Instances in DocRED](https://mlsanity.com/paper/2204.07980/) (2022)
- [Manual Evaluation Matters: Reviewing Test Protocols of Distantly Supervised Relation Extraction](https://mlsanity.com/paper/2105.09543/) (2021)
- [Effective Attention Modeling for Neural Relation Extraction](https://mlsanity.com/paper/1912.03832/) (2019)

## Built on by (most cited first, 30 of 57)

- [Extract, Define, Canonicalize: An LLM-based Framework for Knowledge Graph Construction](https://mlsanity.com/paper/2404.03868/) (2024)
- [DREEAM: Guiding Attention with Evidence for Improving Document-Level Relation Extraction](https://mlsanity.com/paper/2302.08675/) (2023)
- [MemLLM: Finetuning LLMs to Use An Explicit Read-Write Memory](https://mlsanity.com/paper/2404.11672/) (2024)
- [AutoRE: Document-Level Relation Extraction with Large Language Models](https://mlsanity.com/paper/2403.14888/) (2024)
- [Multi-News+: Cost-efficient Dataset Cleansing via LLM-based Data Annotation](https://mlsanity.com/paper/2404.09682/) (2024)
- [Collapse of Dense Retrievers: Short, Early, and Literal Biases Outranking Factual Evidence](https://mlsanity.com/paper/2503.05037/) (2025)
- [Consistency Guided Knowledge Retrieval and Denoising in LLMs for Zero-shot Document-level Relation Triplet Extraction](https://mlsanity.com/paper/2401.13598/) (2024)
- [GLiREL -- Generalist Model for Zero-Shot Relation Extraction](https://mlsanity.com/paper/2501.03172/) (2025)
- [Metamorphic Testing of Large Language Models for Natural Language Processing](https://mlsanity.com/paper/2511.02108/) (2025)
- [Can LLMs be Good Graph Judge for Knowledge Graph Construction?](https://mlsanity.com/paper/2411.17388/) (2024)
- [Semi-automatic Data Enhancement for Document-Level Relation Extraction with Distant Supervision from Large Language Models](https://mlsanity.com/paper/2311.07314/) (2023)
- [WebIE: Faithful and Robust Information Extraction on the Web](https://mlsanity.com/paper/2305.14293/) (2023)
- [LLM with Relation Classifier for Document-Level Relation Extraction](https://mlsanity.com/paper/2408.13889/) (2024)
- [Uncertainty Guided Label Denoising for Document-level Distant Relation Extraction](https://mlsanity.com/paper/2305.11029/) (2023)
- [The Effects of Hallucinations in Synthetic Training Data for Relation Extraction](https://mlsanity.com/paper/2410.08393/) (2024)
- [ATOM: AdapTive and OptiMized dynamic temporal knowledge graph construction using LLMs](https://mlsanity.com/paper/2510.22590/) (2025)
- [Towards Integration of Discriminability and Robustness for Document-Level Relation Extraction](https://mlsanity.com/paper/2304.00824/) (2023)
- [A Frustratingly Easy Improvement for Position Embeddings via Random Padding](https://mlsanity.com/paper/2305.04859/) (2023)
- [A Positive-Unlabeled Metric Learning Framework for Document-Level Relation Extraction with Incomplete Labeling](https://mlsanity.com/paper/2306.14806/) (2023)
- [A Comprehensive Survey of Document-level Relation Extraction (2016-2023)](https://mlsanity.com/paper/2309.16396/) (2023)
- [PromptRE: Weakly-Supervised Document-Level Relation Extraction via Prompting-Based Data Programming](https://mlsanity.com/paper/2310.09265/) (2023)
- [LKD-KGC: Domain-Specific KG Construction via LLM-driven Knowledge Dependency Parsing](https://mlsanity.com/paper/2505.24163/) (2025)
- [A Survey of Document-Level Information Extraction](https://mlsanity.com/paper/2309.13249/) (2023)
- [PRiSM: Enhancing Low-Resource Document-Level Relation Extraction with Relation-Aware Score Calibration](https://mlsanity.com/paper/2309.13869/) (2023)
- [Anaphor Assisted Document-Level Relation Extraction](https://mlsanity.com/paper/2310.18604/) (2023)
- [A Domain-Specific Curated Benchmark for Entity and Document-Level Relation Extraction](https://mlsanity.com/paper/2602.04320/) (2026)
- [Better Sampling of Negatives for Distantly Supervised Named Entity Recognition](https://mlsanity.com/paper/2305.13142/) (2023)
- [REXEL: An End-to-end Model for Document-Level Relation Extraction and Entity Linking](https://mlsanity.com/paper/2404.12788/) (2024)
- [TTM-RE: Memory-Augmented Document-Level Relation Extraction](https://mlsanity.com/paper/2406.05906/) (2024)
- [Combining Supervised Learning and Reinforcement Learning for Multi-Label Classification Tasks with Partial Labels](https://mlsanity.com/paper/2406.16293/) (2024)

## Abstract (arXiv)

The DocRED dataset is one of the most popular and widely used benchmarks for document-level relation extraction (RE). It adopts a recommend-revise annotation scheme so as to have a large-scale annotated dataset. However, we find that the annotation of DocRED is incomplete, i.e., false negative samples are prevalent. We analyze the causes and effects of the overwhelming false negative problem in the DocRED dataset. To address the shortcoming, we re-annotate 4,053 documents in the DocRED dataset by adding the missed relation triples back to the original DocRED. We name our revised DocRED dataset Re-DocRED. We conduct extensive experiments with state-of-the-art neural models on both datasets, and the experimental results show that the models trained and evaluated on our Re-DocRED achieve performance improvements of around 13 F1 points. Moreover, we conduct a comprehensive analysis to identify the potential areas for further improvement. Our dataset is publicly available at https://github.com/tonytan48/Re-DocRED.