# Towards Accurate and Consistent Evaluation: A Dataset for Distantly-Supervised Relation Extraction

arXiv: https://arxiv.org/abs/2010.16275 · HTML: https://mlsanity.com/paper/2010.16275/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 30 Oct 2020.

## Built on by (most cited first, 8 of 8)

- [A Comprehensive Survey on Relation Extraction: Recent Advances and New Frontiers](https://mlsanity.com/paper/2306.02051/) (2023)
- [Multi-hop Evidence Retrieval for Cross-document Relation Extraction](https://mlsanity.com/paper/2212.10786/) (2022)
- [Deep Neural Approaches to Relation Triplets Extraction: A Comprehensive Survey](https://mlsanity.com/paper/2103.16929/) (2021)
- [Manual Evaluation Matters: Reviewing Test Protocols of Distantly Supervised Relation Extraction](https://mlsanity.com/paper/2105.09543/) (2021)
- [DiS-ReX: A Multilingual Dataset for Distantly Supervised Relation Extraction](https://mlsanity.com/paper/2104.08655/) (2021)
- [Distantly Supervised Relation Extraction via Recursive Hierarchy-Interactive Attention and Entity-Order Perception](https://mlsanity.com/paper/2105.08213/) (2021)
- [A systematic review of relation extraction task since the emergence of Transformers](https://mlsanity.com/paper/2511.03610/) (2025)
- [SCOPE and SCION: A Benchmark and an Auditable Reference Pipeline for Schema Induction and Fusion from Text](https://mlsanity.com/paper/2607.21610/) (2026)

## Abstract (arXiv)

In recent years, distantly-supervised relation extraction has achieved a certain success by using deep neural networks. Distant Supervision (DS) can automatically generate large-scale annotated data by aligning entity pairs from Knowledge Bases (KB) to sentences. However, these DS-generated datasets inevitably have wrong labels that result in incorrect evaluation scores during testing, which may mislead the researchers. To solve this problem, we build a new dataset NYTH, where we use the DS-generated data as training data and hire annotators to label test data. Compared with the previous datasets, NYT-H has a much larger test set and then we can perform more accurate and consistent evaluation. Finally, we present the experimental results of several widely used systems on NYT-H. The experimental results show that the ranking lists of the comparison systems on the DS-labelled test data and human-annotated test data are different. This indicates that our human-annotated data is necessary for evaluation of distantly-supervised relation extraction.