# Revisiting the Negative Data of Distantly Supervised Relation Extraction

arXiv: https://arxiv.org/abs/2105.10158 · HTML: https://mlsanity.com/paper/2105.10158/

## Key facts

- Cited by 15 papers in the map (data through 14 Oct 2026).
- Builds on 20 of its 50 references that are in the map.
- First posted on arXiv 21 May 2021.

## Builds on (most cited first, 20 of 20)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [Speech Recognition with Deep Recurrent Neural Networks](https://mlsanity.com/paper/1303.5778/) (2013)
- [Dice Loss for Data-imbalanced NLP Tasks](https://mlsanity.com/paper/1911.02855/) (2019)
- [End-to-End Relation Extraction using LSTMs on Sequences and Tree Structures](https://mlsanity.com/paper/1601.00770/) (2016)
- [Joint Extraction of Entities and Relations Based on a Novel Tagging Scheme](https://mlsanity.com/paper/1706.05075/) (2017)
- [Entity-Relation Extraction as Multi-Turn Question Answering](https://mlsanity.com/paper/1905.05529/) (2019)
- [TPLinker: Single-stage Joint Extraction of Entities and Relations Through Token Pair Linking](https://mlsanity.com/paper/2010.13415/) (2020)
- [A Hierarchical Framework for Relation Extraction with Reinforcement Learning](https://mlsanity.com/paper/1811.03925/) (2018)
- [Machine Comprehension Using Match-LSTM and Answer Pointer](https://mlsanity.com/paper/1608.07905/) (2016)
- [Class-prior Estimation for Learning from Positive and Unlabeled Data](https://mlsanity.com/paper/1611.01586/) (2016)
- [Reinforcement Learning for Relation Classification from Noisy Data](https://mlsanity.com/paper/1808.08013/) (2018)
- [Joint Extraction of Entities and Relations Based on a Novel Decomposition Strategy](https://mlsanity.com/paper/1909.04273/) (2019)
- [Classifying Relations by Ranking with Convolutional Neural Networks](https://mlsanity.com/paper/1504.06580/) (2015)
- [CoType: Joint Extraction of Typed Entities and Relations with Knowledge Bases](https://mlsanity.com/paper/1610.08763/) (2016)
- [Improved Relation Extraction with Feature-Rich Compositional Embedding Models](https://mlsanity.com/paper/1505.02419/) (2015)
- [Improving distant supervision using inference learning](https://mlsanity.com/paper/1509.03739/) (2015)
- [Adaptive Scaling for Sparse Detection in Information Extraction](https://mlsanity.com/paper/1805.00250/) (2018)
- [Collective Loss Function for Positive and Unlabeled Learning](https://mlsanity.com/paper/2005.03228/) (2020)

## Built on by (most cited first, 15 of 15)

- [ChatIE: Zero-Shot Information Extraction via Chatting with ChatGPT](https://mlsanity.com/paper/2302.10205/) (2023)
- [InstructUIE: Multi-task Instruction Tuning for Unified Information Extraction](https://mlsanity.com/paper/2304.08085/) (2023)
- [Does Recommend-Revise Produce Reliable Annotations? An Analysis on Missing Instances in DocRED](https://mlsanity.com/paper/2204.07980/) (2022)
- [What do You Mean by Relation Extraction? A Survey on Datasets and Study on Scientific Relation Classification](https://mlsanity.com/paper/2204.13516/) (2022)
- [R1-RE: Cross-Domain Relation Extraction with RLVR](https://mlsanity.com/paper/2507.04642/) (2025)
- [FastRE: Towards Fast Relation Extraction with Convolutional Encoder and Improved Cascade Binary Tagging Framework](https://mlsanity.com/paper/2205.02490/) (2022)
- [Uncertainty Guided Label Denoising for Document-level Distant Relation Extraction](https://mlsanity.com/paper/2305.11029/) (2023)
- [MultiTACRED: A Multilingual Version of the TAC Relation Extraction Dataset](https://mlsanity.com/paper/2305.04582/) (2023)
- [Improving Recall of Large Language Models: A Model Collaboration Approach for Relational Triple Extraction](https://mlsanity.com/paper/2404.09593/) (2024)
- [Adaptive Reinforcement Learning Planning: Harnessing Large Language Models for Complex Information Extraction](https://mlsanity.com/paper/2406.11455/) (2024)
- [Beyond the Numbers: Transparency in Relation Extraction Benchmark Creation and Leaderboards](https://mlsanity.com/paper/2411.05224/) (2024)
- [Is There a One-Model-Fits-All Approach to Information Extraction? Revisiting Task Definition Biases](https://mlsanity.com/paper/2403.16396/) (2024)
- [Adaptive Ordered Information Extraction with Deep Reinforcement Learning](https://mlsanity.com/paper/2306.10787/) (2023)
- [Few-shot Continual Relation Extraction via Open Information Extraction](https://mlsanity.com/paper/2502.16648/) (2025)
- [RLAP: A Reinforcement Learning Enhanced Adaptive Planning Framework for Multi-step NLP Task Solving](https://mlsanity.com/paper/2505.11893/) (2025)

## Abstract (arXiv)

Distantly supervision automatically generates plenty of training samples for relation extraction. However, it also incurs two major problems: noisy labels and imbalanced training data. Previous works focus more on reducing wrongly labeled relations (false positives) while few explore the missing relations that are caused by incompleteness of knowledge base (false negatives). Furthermore, the quantity of negative labels overwhelmingly surpasses the positive ones in previous problem formulations. In this paper, we first provide a thorough analysis of the above challenges caused by negative data. Next, we formulate the problem of relation extraction into as a positive unlabeled learning task to alleviate false negative problem. Thirdly, we propose a pipeline approach, dubbed textscReRe, that performs sentence-level relation detection then subject/object extraction to achieve sample-efficient training. Experimental results show that the proposed method consistently outperforms existing approaches and remains excellent performance even learned with a large quantity of false positive samples.