mlsanityData to 7 Oct 2026

Revisiting DocRED -- Addressing the False Negative Problem in Relation Extraction

arXiv 2205.12696 ↗First posted 25 May 2022cs.CL cs.IRcited by 57

Not on the map. Why?

Abstract (arXiv)

The DocRED dataset is one of the most popular and widely used benchmarks for document-level relation extraction (RE). It adopts a recommend-revise annotation scheme so as to have a large-scale annotated dataset. However, we find that the annotation of DocRED is incomplete, i.e., false negative samples are prevalent. We analyze the causes and effects of the overwhelming false negative problem in the DocRED dataset. To address the shortcoming, we re-annotate 4,053 documents in the DocRED dataset by adding the missed relation triples back to the original DocRED. We name our revised DocRED dataset Re-DocRED. We conduct extensive experiments with state-of-the-art neural models on both datasets, and the experimental results show that the models trained and evaluated on our Re-DocRED achieve performance improvements of around 13 F1 points. Moreover, we conduct a comprehensive analysis to identify the potential areas for further improvement. Our dataset is publicly available at https://github.com/tonytan48/Re-DocRED.

Key facts. Cited by 57 papers in the map (data through 7 Oct 2026). Builds on 32 of its 54 references that are in the map. First posted on arXiv 25 May 2022.

Builds on

32 of 54 references.

References · 32

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  4. 2019Unsupervised Cross-lingual Representation Learning at Scalecited by 3,907
  5. 2021Symbolic Knowledge Distillation: from General Language Models to Commonsense Modelscited by 270
  6. 2018FewRel: A Large-Scale Supervised Few-Shot Relation Classification Dataset with State-of-the-Art Evaluationcited by 254
  7. 2019DocRED: A Large-Scale Document-Level Relation Extraction Datasetcited by 239
  8. 2021Generating Datasets with Pretrained Language Modelscited by 192
  9. 2019Not Enough Data? Deep Learning to the Rescue!cited by 174
  10. 2020Data Augmentation using Pre-trained Transformer Modelscited by 168
  11. 2020Document-Level Relation Extraction with Adaptive Thresholding and Localized Context Poolingcited by 115
  12. 2020Reasoning with Latent Structure Refinement for Document-Level Relation Extractioncited by 97
20 more
  1. 2020TACRED Revisited: A Thorough Evaluation of the TACRED Relation Extraction Taskcited by 96
  2. 2019FewRel 2.0: Towards More Challenging Few-Shot Relation Classificationcited by 94
  3. 2021An Improved Baseline for Sentence-level Relation Extractioncited by 80
  4. 2021Document-level Relation Extraction as Semantic Segmentationcited by 80
  5. 2019Effective Modeling of Encoder-Decoder Architecture for Joint Entity and Relation Extractioncited by 73
  6. 2021Re-TACRED: Addressing Shortcomings of the TACRED Datasetcited by 73
  7. 2019SemEval-2010 Task 8: Multi-Way Classification of Semantic Relations Between Pairs of Nominalscited by 63
  8. 2022RelationPrompt: Leveraging Prompts to Generate Synthetic Data for Zero-Shot Relation Triplet Extractioncited by 60
  9. 2020More Data, More Relations, More Context and More Openness: A Review and Outlook for Relation Extractioncited by 52
  10. 2021MELM: Data Augmentation with Masked Entity Language Modeling for Low-Resource NERcited by 42
  11. 2022Document-Level Relation Extraction with Adaptive Focal Loss and Knowledge Distillationcited by 39
  12. 2020Empirical Analysis of Unlabeled Entity Problem in Named Entity Recognitioncited by 34
  13. 2021An End-to-end Model for Entity-level Relation Extraction using Multi-instance Learningcited by 31
  14. 2019Tackling Long-Tailed Relations and Uncommon Entities in Knowledge Graph Completioncited by 19
  15. 2020Denoising Relation Extraction from Document-level Distant Supervisioncited by 19
  16. 2022Does Recommend-Revise Produce Reliable Annotations? An Analysis on Missing Instances in DocREDcited by 19
  17. 2021Manual Evaluation Matters: Reviewing Test Protocols of Distantly Supervised Relation Extractioncited by 12
  18. 2019Effective Attention Modeling for Neural Relation Extractioncited by 10
  19. 2021Knowing False Negatives: An Adversarial Training Method for Distantly Supervised Relation Extractioncited by 5
  20. 2020H-FND: Hierarchical False-Negative Denoising for Distant Supervision Relation Extractioncited by 3

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 14; 2024: 19; 2025: 13; 2026: 11.

57 papers.

Sort:

By area:

  1. 2024Extract, Define, Canonicalize: An LLM-based Framework for Knowledge Graph Constructioncited by 70
  2. 2023DREEAM: Guiding Attention with Evidence for Improving Document-Level Relation Extractioncited by 37
  3. 2024MemLLM: Finetuning LLMs to Use An Explicit Read-Write Memorycited by 35
  4. 2024AutoRE: Document-Level Relation Extraction with Large Language Modelscited by 16
  5. 2024Multi-News+: Cost-efficient Dataset Cleansing via LLM-based Data Annotationcited by 16
  6. 2025Collapse of Dense Retrievers: Short, Early, and Literal Biases Outranking Factual Evidencecited by 16
  7. 2024Consistency Guided Knowledge Retrieval and Denoising in LLMs for Zero-shot Document-level Relation Triplet Extractioncited by 14
  8. 2025Metamorphic Testing of Large Language Models for Natural Language Processingcited by 12
  9. 2024Can LLMs be Good Graph Judge for Knowledge Graph Construction?cited by 11
  10. 2025GLiREL -- Generalist Model for Zero-Shot Relation Extractioncited by 11
  11. 2023Semi-automatic Data Enhancement for Document-Level Relation Extraction with Distant Supervision from Large Language Modelscited by 9
  12. 2023WebIE: Faithful and Robust Information Extraction on the Webcited by 8
38 more
  1. 2024LLM with Relation Classifier for Document-Level Relation Extractioncited by 8
  2. 2023Uncertainty Guided Label Denoising for Document-level Distant Relation Extractioncited by 7
  3. 2024The Effects of Hallucinations in Synthetic Training Data for Relation Extractioncited by 7
  4. 2025ATOM: AdapTive and OptiMized dynamic temporal knowledge graph construction using LLMscited by 7
  5. 2023Towards Integration of Discriminability and Robustness for Document-Level Relation Extractioncited by 4
  6. 2023A Frustratingly Easy Improvement for Position Embeddings via Random Paddingcited by 4
  7. 2023A Positive-Unlabeled Metric Learning Framework for Document-Level Relation Extraction with Incomplete Labelingcited by 4
  8. 2023A Comprehensive Survey of Document-level Relation Extraction (2016-2023)cited by 4
  9. 2023PromptRE: Weakly-Supervised Document-Level Relation Extraction via Prompting-Based Data Programmingcited by 4
  10. 2025LKD-KGC: Domain-Specific KG Construction via LLM-driven Knowledge Dependency Parsingcited by 4
  11. 2023PRiSM: Enhancing Low-Resource Document-Level Relation Extraction with Relation-Aware Score Calibrationcited by 3
  12. 2023Anaphor Assisted Document-Level Relation Extractioncited by 3
  13. 2026A Domain-Specific Curated Benchmark for Entity and Document-Level Relation Extractioncited by 3
  14. 2023Better Sampling of Negatives for Distantly Supervised Named Entity Recognitioncited by 2
  15. 2023A Survey of Document-Level Information Extractioncited by 2
  16. 2024REXEL: An End-to-end Model for Document-Level Relation Extraction and Entity Linkingcited by 2
  17. 2024TTM-RE: Memory-Augmented Document-Level Relation Extractioncited by 2
  18. 2024Combining Supervised Learning and Reinforcement Learning for Multi-Label Classification Tasks with Partial Labelscited by 2
  19. 2025A Benchmark for End-to-End Zero-Shot Biomedical Relation Extraction with LLMs: Experiments with OpenAI Modelscited by 2
  20. 2026CLEF HIPE-2026: Evaluating Accurate and Efficient Person-Place Relation Extraction from Multilingual Historical Textscited by 2
  21. 2023GADePo: Graph-Assisted Declarative Pooling Transformers for Document-Level Relation Extractioncited by 1
  22. 2024GEGA: Graph Convolutional Networks and Evidence Retrieval Guided Attention for Enhanced Document-level Relation Extractioncited by 1
  23. 2025DocIE@XLLM25: In-Context Learning for Information Extraction using Fully Synthetic Demonstrationscited by 1
  24. 2025A systematic review of relation extraction task since the emergence of Transformerscited by 1
  25. 2024READ: Improving Relation Extraction from an ADversarial Perspectivecited by 0
  26. 2024Building a Japanese Document-Level Relation Extraction Dataset Assisted by Cross-Lingual Transfercited by 0
  27. 2024Maximizing Relation Extraction Potential: A Data-Centric Study to Unveil Challenges and Opportunitiescited by 0
  28. 2024DiVA-DocRE: A Discriminative and Voice-Aware Paradigm for Document-Level Relation Extractioncited by 0
  29. 2024Graph-DPEP: Decomposed Plug and Ensemble Play for Few-Shot Document Relation Extraction with Graph-of-Thoughts Reasoningcited by 0
  30. 2024VaeDiff-DocRE: End-to-end Data Augmentation Framework for Document-level Relation Extractioncited by 0
  31. 2024KnowRA: Knowledge Retrieval Augmented Method for Document-level Relation Extraction with Comprehensive Reasoning Abilitiescited by 0
  32. 2025Interim Report on Human-Guided Adaptive Hyperparameter Optimization with Multi-Fidelity Sprintscited by 0
  33. 2025Analyzing the Influence of Knowledge Graph Information on Relation Extractioncited by 0
  34. 2025EMERGE: A Benchmark for Updating Knowledge Graphs with Emerging Textual Knowledgecited by 0
  35. 2025Multi-Relation Extraction in Entity Pairs using Global Contextcited by 0
  36. 2025Relation as a Prior: A Novel Paradigm for LLM-based Document-level Relation Extractioncited by 0
  37. 2026Document-Level Zero-Shot Relation Extraction with Entity Side Informationcited by 0
  38. 2026DOREMI: Optimizing Long Tail Predictions in Document-Level Relation Extractioncited by 0

50 most cited of 57 All 57 (JSON)