# Does Recommend-Revise Produce Reliable Annotations? An Analysis on Missing Instances in DocRED

arXiv: https://arxiv.org/abs/2204.07980 · HTML: https://mlsanity.com/paper/2204.07980/

## Key facts

- Cited by 19 papers in the map (data through 14 Oct 2026).
- Builds on 15 of its 24 references that are in the map.
- First posted on arXiv 17 Apr 2022.

## Builds on (most cited first, 15 of 15)

- [Pervasive Label Errors in Test Sets Destabilize Machine Learning Benchmarks](https://mlsanity.com/paper/2103.14749/) (2021)
- [DocRED: A Large-Scale Document-Level Relation Extraction Dataset](https://mlsanity.com/paper/1906.06127/) (2019)
- [Graph Convolution over Pruned Dependency Trees Improves Relation Extraction](https://mlsanity.com/paper/1809.10185/) (2018)
- [Document-Level Relation Extraction with Adaptive Thresholding and Localized Context Pooling](https://mlsanity.com/paper/2010.11304/) (2020)
- [Cross-Sentence N-ary Relation Extraction with Graph LSTMs](https://mlsanity.com/paper/1708.03743/) (2017)
- [Reasoning with Latent Structure Refinement for Document-Level Relation Extraction](https://mlsanity.com/paper/2005.06312/) (2020)
- [TACRED Revisited: A Thorough Evaluation of the TACRED Relation Extraction Task](https://mlsanity.com/paper/2004.14855/) (2020)
- [Double Graph Based Reasoning for Document-level Relation Extraction](https://mlsanity.com/paper/2009.13752/) (2020)
- [Document-level Relation Extraction as Semantic Segmentation](https://mlsanity.com/paper/2106.03618/) (2021)
- [Re-TACRED: Addressing Shortcomings of the TACRED Dataset](https://mlsanity.com/paper/2104.08398/) (2021)
- [Entity Structure Within and Throughout: Modeling Mention Dependencies for Document-Level Relation Extraction](https://mlsanity.com/paper/2102.10249/) (2021)
- [Distantly Supervised Named Entity Recognition using Positive-Unlabeled Learning](https://mlsanity.com/paper/1906.01378/) (2019)
- [Three Sentences Are All You Need: Local Path Enhanced Document Relation Extraction](https://mlsanity.com/paper/2106.01793/) (2021)
- [Revisiting the Negative Data of Distantly Supervised Relation Extraction](https://mlsanity.com/paper/2105.10158/) (2021)
- [Improving distant supervision using inference learning](https://mlsanity.com/paper/1509.03739/) (2015)

## Built on by (most cited first, 19 of 19)

- [Revisiting DocRED -- Addressing the False Negative Problem in Relation Extraction](https://mlsanity.com/paper/2205.12696/) (2022)
- [DREEAM: Guiding Attention with Evidence for Improving Document-Level Relation Extraction](https://mlsanity.com/paper/2302.08675/) (2023)
- [Position: Measure Dataset Diversity, Don't Just Claim It](https://mlsanity.com/paper/2407.08188/) (2024)
- [A Unified Positive-Unlabeled Learning Framework for Document-Level Relation Extraction with Different Levels of Labeling](https://mlsanity.com/paper/2210.08709/) (2022)
- [RAPL: A Relation-Aware Prototype Learning Approach for Few-Shot Document-Level Relation Extraction](https://mlsanity.com/paper/2310.15743/) (2023)
- [Semi-automatic Data Enhancement for Document-Level Relation Extraction with Distant Supervision from Large Language Models](https://mlsanity.com/paper/2311.07314/) (2023)
- [Towards Integration of Discriminability and Robustness for Document-Level Relation Extraction](https://mlsanity.com/paper/2304.00824/) (2023)
- [A Positive-Unlabeled Metric Learning Framework for Document-Level Relation Extraction with Incomplete Labeling](https://mlsanity.com/paper/2306.14806/) (2023)
- [A Comprehensive Survey of Document-level Relation Extraction (2016-2023)](https://mlsanity.com/paper/2309.16396/) (2023)
- [Did the Models Understand Documents? Benchmarking Models for Language Understanding in Document-Level Relation Extraction](https://mlsanity.com/paper/2306.11386/) (2023)
- [Anaphor Assisted Document-Level Relation Extraction](https://mlsanity.com/paper/2310.18604/) (2023)
- [Class-Adaptive Self-Training for Relation Extraction with Incompletely Annotated Training Data](https://mlsanity.com/paper/2306.09697/) (2023)
- [Combining Supervised Learning and Reinforcement Learning for Multi-Label Classification Tasks with Partial Labels](https://mlsanity.com/paper/2406.16293/) (2024)
- [On the Robustness of Document-Level Relation Extraction Models to Entity Name Variations](https://mlsanity.com/paper/2406.07444/) (2024)
- [GEGA: Graph Convolutional Networks and Evidence Retrieval Guided Attention for Enhanced Document-level Relation Extraction](https://mlsanity.com/paper/2407.21384/) (2024)
- [TPN: Transferable Proto-Learning Network towards Few-shot Document-Level Relation Extraction](https://mlsanity.com/paper/2410.00412/) (2024)
- [Building a Japanese Document-Level Relation Extraction Dataset Assisted by Cross-Lingual Transfer](https://mlsanity.com/paper/2404.16506/) (2024)
- [DOREMI: Optimizing Long Tail Predictions in Document-Level Relation Extraction](https://mlsanity.com/paper/2601.11190/) (2026)
- [Ontology-Driven Structural Regularization for Document-Level Relation Extraction](https://mlsanity.com/paper/2608.20856/) (2026)

## Abstract (arXiv)

DocRED is a widely used dataset for document-level relation extraction. In the large-scale annotation, a textitrecommend-revise scheme is adopted to reduce the workload. Within this scheme, annotators are provided with candidate relation instances from distant supervision, and they then manually supplement and remove relational facts based on the recommendations. However, when comparing DocRED with a subset relabeled from scratch, we find that this scheme results in a considerable amount of false negative samples and an obvious bias towards popular entities and relations. Furthermore, we observe that the models trained on DocRED have low recall on our relabeled dataset and inherit the same bias in the training data. Through the analysis of annotators' behaviors, we figure out the underlying reason for the problems above: the scheme actually discourages annotators from supplementing adequate instances in the revision phase. We appeal to future research to take into consideration the issues with the recommend-revise scheme when designing new models and annotation schemes. The relabeled dataset is released at urlhttps://github.com/AndrewZhe/Revisit-DocRED, to serve as a more reliable test set of document RE models.