# An Overview of Distant Supervision for Relation Extraction with a Focus on Denoising and Pre-training Methods

arXiv: https://arxiv.org/abs/2207.08286 · HTML: https://mlsanity.com/paper/2207.08286/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 29 of its 52 references that are in the map.
- First posted on arXiv 17 Jul 2022.

## Builds on (most cited first, 29 of 29)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [Distributed Representations of Words and Phrases and their Compositionality](https://mlsanity.com/paper/1310.4546/) (2013)
- [SciBERT: A Pretrained Language Model for Scientific Text](https://mlsanity.com/paper/1903.10676/) (2019)
- [Domain-Specific Language Model Pretraining for Biomedical Natural Language Processing](https://mlsanity.com/paper/2007.15779/) (2020)
- [Matching the Blanks: Distributional Similarity for Relation Learning](https://mlsanity.com/paper/1906.03158/) (2019)
- [FewRel: A Large-Scale Supervised Few-Shot Relation Classification Dataset with State-of-the-Art Evaluation](https://mlsanity.com/paper/1810.10147/) (2018)
- [DocRED: A Large-Scale Document-Level Relation Extraction Dataset](https://mlsanity.com/paper/1906.06127/) (2019)
- [End-to-End Relation Extraction using LSTMs on Sequences and Tree Structures](https://mlsanity.com/paper/1601.00770/) (2016)
- [Cross-Sentence N-ary Relation Extraction with Graph LSTMs](https://mlsanity.com/paper/1708.03743/) (2017)
- [FewRel 2.0: Towards More Challenging Few-Shot Relation Classification](https://mlsanity.com/paper/1910.07124/) (2019)
- [Learning from Context or Names? An Empirical Study on Neural Relation Extraction](https://mlsanity.com/paper/2010.01923/) (2020)
- [ERICA: Improving Entity and Relation Understanding for Pre-trained Language Models via Contrastive Learning](https://mlsanity.com/paper/2012.15022/) (2020)
- [SemEval-2010 Task 8: Multi-Way Classification of Semantic Relations Between Pairs of Nominals](https://mlsanity.com/paper/1911.10422/) (2019)
- [Distant Supervision for Relation Extraction beyond the Sentence Boundary](https://mlsanity.com/paper/1609.04873/) (2016)
- [Relation Classification via Recurrent Neural Network](https://mlsanity.com/paper/1508.01006/) (2015)
- [Fine-tune Bert for DocRED with Two-step Process](https://mlsanity.com/paper/1909.11898/) (2019)
- [HIN: Hierarchical Inference Network for Document-Level Relation Extraction](https://mlsanity.com/paper/2003.12754/) (2020)
- [Question Answering on Freebase via Relation Extraction and Textual Evidence](https://mlsanity.com/paper/1603.00957/) (2016)
- [Fine-tuning Pre-Trained Transformer Language Models to Distantly Supervised Relation Extraction](https://mlsanity.com/paper/1906.08646/) (2019)
- [Denoising Relation Extraction from Document-level Distant Supervision](https://mlsanity.com/paper/2011.03888/) (2020)
- [Self-Attention Enhanced Selective Gate with Entity-Aware Embedding for Distantly Supervised Relation Extraction](https://mlsanity.com/paper/1911.11899/) (2019)
- [Learning with Noise: Enhance Distantly Supervised Relation Extraction with Dynamic Transition Matrix](https://mlsanity.com/paper/1705.03995/) (2017)
- [Manual Evaluation Matters: Reviewing Test Protocols of Distantly Supervised Relation Extraction](https://mlsanity.com/paper/2105.09543/) (2021)
- [A Data-driven Approach for Noise Reduction in Distantly Supervised Biomedical Relation Extraction](https://mlsanity.com/paper/2005.12565/) (2020)
- [How Knowledge Graph and Attention Help? A Quantitative Analysis into Bag-level Relation Extraction](https://mlsanity.com/paper/2107.12064/) (2021)
- [Abstractified Multi-instance Learning (AMIL) for Biomedical Relation Extraction](https://mlsanity.com/paper/2110.12501/) (2021)
- [Fine-grained Contrastive Learning for Relation Extraction](https://mlsanity.com/paper/2205.12491/) (2022)

## Built on by (most cited first, 3 of 3)

- [READ: Improving Relation Extraction from an ADversarial Perspective](https://mlsanity.com/paper/2404.02931/) (2024)
- [Towards DS-NER: Unveiling and Addressing Latent Noise in Distant Annotations](https://mlsanity.com/paper/2505.12454/) (2025)
- [SCoRE: Streamlined Corpus-based Relation Extraction using Multi-Label Contrastive Learning and Bayesian kNN](https://mlsanity.com/paper/2507.06895/) (2025)

## Abstract (arXiv)

Relation Extraction (RE) is a foundational task of natural language processing. RE seeks to transform raw, unstructured text into structured knowledge by identifying relational information between entity pairs found in text. RE has numerous uses, such as knowledge graph completion, text summarization, question-answering, and search querying. The history of RE methods can be roughly organized into four phases: pattern-based RE, statistical-based RE, neural-based RE, and large language model-based RE. This survey begins with an overview of a few exemplary works in the earlier phases of RE, highlighting limitations and shortcomings to contextualize progress. Next, we review popular benchmarks and critically examine metrics used to assess RE performance. We then discuss distant supervision, a paradigm that has shaped the development of modern RE methods. Lastly, we review recent RE works focusing on denoising and pre-training methods.