# Active Testing: An Unbiased Evaluation Method for Distantly Supervised Relation Extraction

arXiv: https://arxiv.org/abs/2010.08777 · HTML: https://mlsanity.com/paper/2010.08777/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 17 Oct 2020.

## Built on by (most cited first, 3 of 3)

- [Manual Evaluation Matters: Reviewing Test Protocols of Distantly Supervised Relation Extraction](https://mlsanity.com/paper/2105.09543/) (2021)
- [Mention Annotations Alone Enable Efficient Domain Adaptation for Coreference Resolution](https://mlsanity.com/paper/2210.07602/) (2022)
- [Active Real-World Factor-Based Evaluation for Generalist Robot Policies](https://mlsanity.com/paper/2607.14439/) (2026)

## Abstract (arXiv)

Distant supervision has been a widely used method for neural relation extraction for its convenience of automatically labeling datasets. However, existing works on distantly supervised relation extraction suffer from the low quality of test set, which leads to considerable biased performance evaluation. These biases not only result in unfair evaluations but also mislead the optimization of neural relation extraction. To mitigate this problem, we propose a novel evaluation method named active testing through utilizing both the noisy test set and a few manual annotations. Experiments on a widely used benchmark show that our proposed approach can yield approximately unbiased evaluations for distantly supervised relation extractors.