# Manual Evaluation Matters: Reviewing Test Protocols of Distantly Supervised Relation Extraction arXiv: https://arxiv.org/abs/2105.09543 ยท HTML: https://mlsanity.com/paper/2105.09543/ ## Key facts - Cited by 12 papers in the map (data through 14 Oct 2026). - Builds on 24 of its 52 references that are in the map. - First posted on arXiv 20 May 2021. ## Builds on (most cited first, 24 of 24) - [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018) - [Language Models as Knowledge Bases?](https://mlsanity.com/paper/1909.01066/) (2019) - [Reading Wikipedia to Answer Open-Domain Questions](https://mlsanity.com/paper/1704.00051/) (2017) - [ERNIE: Enhanced Language Representation with Informative Entities](https://mlsanity.com/paper/1905.07129/) (2019) - [Matching the Blanks: Distributional Similarity for Relation Learning](https://mlsanity.com/paper/1906.03158/) (2019) - [Large-scale Simple Question Answering with Memory Networks](https://mlsanity.com/paper/1506.02075/) (2015) - [Cross-Sentence N-ary Relation Extraction with Graph LSTMs](https://mlsanity.com/paper/1708.03743/) (2017) - [Learning from Context or Names? An Empirical Study on Neural Relation Extraction](https://mlsanity.com/paper/2010.01923/) (2020) - [Distant Supervision for Relation Extraction beyond the Sentence Boundary](https://mlsanity.com/paper/1609.04873/) (2016) - [OpenNRE: An Open and Extensible Toolkit for Neural Relation Extraction](https://mlsanity.com/paper/1909.13078/) (2019) - [Long-tail Relation Extraction via Knowledge Graph Embeddings and Graph Convolution Networks](https://mlsanity.com/paper/1903.01306/) (2019) - [More Data, More Relations, More Context and More Openness: A Review and Outlook for Relation Extraction](https://mlsanity.com/paper/2004.03186/) (2020) - [RESIDE: Improving Distantly-Supervised Neural Relation Extraction using Side Information](https://mlsanity.com/paper/1812.04361/) (2018) - [Relation Classification via Recurrent Neural Network](https://mlsanity.com/paper/1508.01006/) (2015) - [Reinforcement Learning for Relation Classification from Noisy Data](https://mlsanity.com/paper/1808.08013/) (2018) - [Fine-tuning Pre-Trained Transformer Language Models to Distantly Supervised Relation Extraction](https://mlsanity.com/paper/1906.08646/) (2019) - [Improving Distantly Supervised Relation Extraction using Word and Entity Based Attention](https://mlsanity.com/paper/1804.06987/) (2018) - [Robust Distant Supervision Relation Extraction via Deep Reinforcement Learning](https://mlsanity.com/paper/1805.09927/) (2018) - [Graph Neural Networks with Generated Parameters for Relation Extraction](https://mlsanity.com/paper/1902.00756/) (2019) - [Multilingual Relation Extraction using Compositional Universal Schema](https://mlsanity.com/paper/1511.06396/) (2015) - [A Data-driven Approach for Noise Reduction in Distantly Supervised Biomedical Relation Extraction](https://mlsanity.com/paper/2005.12565/) (2020) - [Towards Accurate and Consistent Evaluation: A Dataset for Distantly-Supervised Relation Extraction](https://mlsanity.com/paper/2010.16275/) (2020) - [Active Testing: An Unbiased Evaluation Method for Distantly Supervised Relation Extraction](https://mlsanity.com/paper/2010.08777/) (2020) - [Denoising Distant Supervision for Relation Extraction via Instance-Level Adversarial Training](https://mlsanity.com/paper/1805.10959/) (2018) ## Built on by (most cited first, 12 of 12) - [Revisiting DocRED -- Addressing the False Negative Problem in Relation Extraction](https://mlsanity.com/paper/2205.12696/) (2022) - [Fine-grained Contrastive Learning for Relation Extraction](https://mlsanity.com/paper/2205.12491/) (2022) - [Plug-and-Play Knowledge Injection for Pre-trained Language Models](https://mlsanity.com/paper/2305.17691/) (2023) - [Few-Shot Cross-lingual Transfer for Coarse-grained De-identification of Code-Mixed Clinical Texts](https://mlsanity.com/paper/2204.04775/) (2022) - [PARE: A Simple and Strong Baseline for Monolingual and Multilingual Distantly Supervised Relation Extraction](https://mlsanity.com/paper/2110.07415/) (2021) - [An Overview of Distant Supervision for Relation Extraction with a Focus on Denoising and Pre-training Methods](https://mlsanity.com/paper/2207.08286/) (2022) - [MedDistant19: Towards an Accurate Benchmark for Broad-Coverage Biomedical Relation Extraction](https://mlsanity.com/paper/2204.04779/) (2022) - [COMBO: A Complete Benchmark for Open KG Canonicalization](https://mlsanity.com/paper/2302.03905/) (2023) - [Cross-stitching Text and Knowledge Graph Encoders for Distantly Supervised Relation Extraction](https://mlsanity.com/paper/2211.01432/) (2022) - [Knowledge-Enhanced Relation Extraction Dataset](https://mlsanity.com/paper/2210.11231/) (2022) - [SCoRE: Streamlined Corpus-based Relation Extraction using Multi-Label Contrastive Learning and Bayesian kNN](https://mlsanity.com/paper/2507.06895/) (2025) - [Combining Distantly Supervised Models with In Context Learning for Monolingual and Cross-Lingual Relation Extraction](https://mlsanity.com/paper/2510.18344/) (2025) ## Abstract (arXiv) Distantly supervised (DS) relation extraction (RE) has attracted much attention in the past few years as it can utilize large-scale auto-labeled data. However, its evaluation has long been a problem: previous works either took costly and inconsistent methods to manually examine a small sample of model predictions, or directly test models on auto-labeled data -- which, by our check, produce as much as 53% wrong labels at the entity pair level in the popular NYT10 dataset. This problem has not only led to inaccurate evaluation, but also made it hard to understand where we are and what's left to improve in the research of DS-RE. To evaluate DS-RE models in a more credible way, we build manually-annotated test sets for two DS-RE datasets, NYT10 and Wiki20, and thoroughly evaluate several competitive models, especially the latest pre-trained ones. The experimental results show that the manual evaluation can indicate very different conclusions from automatic ones, especially some unexpected observations, e.g., pre-trained models can achieve dominating performance while being more susceptible to false-positives compared to previous methods. We hope that both our manual test sets and novel observations can help advance future DS-RE research.