# Improving Distantly-Supervised Relation Extraction through BERT-based Label & Instance Embeddings

arXiv: https://arxiv.org/abs/2102.01156 · HTML: https://mlsanity.com/paper/2102.01156/

## Key facts

- Cited by 10 papers in the map (data through 14 Oct 2026).
- Builds on 15 of its 34 references that are in the map.
- First posted on arXiv 1 Feb 2021.

## Builds on (most cited first, 15 of 15)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Neural Machine Translation of Rare Words with Subword Units](https://mlsanity.com/paper/1508.07909/) (2015)
- [Simple BERT Models for Relation Extraction and Semantic Role Labeling](https://mlsanity.com/paper/1904.05255/) (2019)
- [Joint Embedding of Words and Labels for Text Classification](https://mlsanity.com/paper/1805.04174/) (2018)
- [RESIDE: Improving Distantly-Supervised Neural Relation Extraction using Side Information](https://mlsanity.com/paper/1812.04361/) (2018)
- [Fine-tuning Pre-Trained Transformer Language Models to Distantly Supervised Relation Extraction](https://mlsanity.com/paper/1906.08646/) (2019)
- [Classifying Relations via Long Short Term Memory Networks along Shortest Dependency Path](https://mlsanity.com/paper/1508.03720/) (2015)
- [Improving Distantly Supervised Relation Extraction using Word and Entity Based Attention](https://mlsanity.com/paper/1804.06987/) (2018)
- [Distant Supervision Relation Extraction with Intra-Bag and Inter-Bag Attentions](https://mlsanity.com/paper/1904.00143/) (2019)
- [DSGAN: Generative Adversarial Training for Distant Supervision Relation Extraction](https://mlsanity.com/paper/1805.09929/) (2018)
- [SEE: Syntax-aware Entity Embedding for Neural Relation Extraction](https://mlsanity.com/paper/1801.03603/) (2018)
- [Neural Relation Extraction via Inner-Sentence Noise Reduction and Transfer Learning](https://mlsanity.com/paper/1808.06738/) (2018)
- [Distantly-Supervised Neural Relation Extraction with Side Information using BERT](https://mlsanity.com/paper/2004.14443/) (2020)

## Built on by (most cited first, 10 of 10)

- [Zero-shot Referring Expression Comprehension via Structural Similarity Between Images and Captions](https://mlsanity.com/paper/2311.17048/) (2023)
- [CIL: Contrastive Instance Learning Framework for Distantly Supervised Relation Extraction](https://mlsanity.com/paper/2106.10855/) (2021)
- [HiCLRE: A Hierarchical Contrastive Learning Framework for Distantly Supervised Relation Extraction](https://mlsanity.com/paper/2202.13352/) (2022)
- [Improving Zero-Shot Entity Retrieval through Effective Dense Representations](https://mlsanity.com/paper/2103.04156/) (2021)
- [PARE: A Simple and Strong Baseline for Monolingual and Multilingual Distantly Supervised Relation Extraction](https://mlsanity.com/paper/2110.07415/) (2021)
- [Sentence Bag Graph Formulation for Biomedical Distant Supervision Relation Extraction](https://mlsanity.com/paper/2310.18912/) (2023)
- [A systematic review of relation extraction task since the emergence of Transformers](https://mlsanity.com/paper/2511.03610/) (2025)
- [Comparative Analysis of Contextual Relation Extraction based on Deep Learning Models](https://mlsanity.com/paper/2309.06814/) (2023)
- [SCoRE: Streamlined Corpus-based Relation Extraction using Multi-Label Contrastive Learning and Bayesian kNN](https://mlsanity.com/paper/2507.06895/) (2025)
- [Sub-Billion, Super-Frontier: Small Language Models Rival Zero-Shot Frontier LLMs on General and Literary Relation Extraction](https://mlsanity.com/paper/2606.22606/) (2026)

## Abstract (arXiv)

Distantly-supervised relation extraction (RE) is an effective method to scale RE to large corpora but suffers from noisy labels. Existing approaches try to alleviate noise through multi-instance learning and by providing additional information, but manage to recognize mainly the top frequent relations, neglecting those in the long-tail. We propose REDSandT (Relation Extraction with Distant Supervision and Transformers), a novel distantly-supervised transformer-based RE method, that manages to capture a wider set of relations through highly informative instance and label embeddings for RE, by exploiting BERT's pre-trained model, and the relationship between labels and entities, respectively. We guide REDSandT to focus solely on relational tokens by fine-tuning BERT on a structured input, including the sub-tree connecting an entity pair and the entities' types. Using the extracted informative vectors, we shape label embeddings, which we also use as attention mechanism over instances to further reduce noise. Finally, we represent sentences by concatenating relation and instance embeddings. Experiments in the NYT-10 dataset show that REDSandT captures a broader set of relations with higher confidence, achieving state-of-the-art AUC (0.424).