# Fine-tuning Pre-Trained Transformer Language Models to Distantly Supervised Relation Extraction

arXiv: https://arxiv.org/abs/1906.08646 · HTML: https://mlsanity.com/paper/1906.08646/

## Key facts

- Cited by 36 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 19 Jun 2019.

## Built on by (most cited first, 30 of 36)

- [MentalBERT: Publicly Available Pretrained Language Models for Mental Healthcare](https://mlsanity.com/paper/2110.15621/) (2021)
- [An Improved Baseline for Sentence-level Relation Extraction](https://mlsanity.com/paper/2102.01373/) (2021)
- [LasUIE: Unifying Information Extraction with Latent Adaptive Structure-aware Generative Language Model](https://mlsanity.com/paper/2304.06248/) (2023)
- [Selecting Large Language Model to Fine-tune via Rectified Scaling Law](https://mlsanity.com/paper/2402.02314/) (2024)
- [Multi-scale Matching Networks for Semantic Correspondence](https://mlsanity.com/paper/2108.00211/) (2021)
- [SPDF: Sparse Pre-training and Dense Fine-tuning for Large Language Models](https://mlsanity.com/paper/2303.10464/) (2023)
- [Crossing the Conversational Chasm: A Primer on Natural Language Processing for Multilingual Task-Oriented Dialogue Systems](https://mlsanity.com/paper/2104.08570/) (2021)
- [Improving Retrieval-Augmented Large Language Models via Data Importance Learning](https://mlsanity.com/paper/2307.03027/) (2023)
- [Billion-scale Pre-trained E-commerce Product Knowledge Graph Model](https://mlsanity.com/paper/2105.00388/) (2021)
- [CIL: Contrastive Instance Learning Framework for Distantly Supervised Relation Extraction](https://mlsanity.com/paper/2106.10855/) (2021)
- [Manual Evaluation Matters: Reviewing Test Protocols of Distantly Supervised Relation Extraction](https://mlsanity.com/paper/2105.09543/) (2021)
- [Improving Distantly-Supervised Relation Extraction through BERT-based Label & Instance Embeddings](https://mlsanity.com/paper/2102.01156/) (2021)
- [SENT: Sentence-level Distant Relation Extraction via Negative Training](https://mlsanity.com/paper/2106.11566/) (2021)
- [Distantly-Supervised Long-Tailed Relation Extraction Using Constraint Graphs](https://mlsanity.com/paper/2105.11225/) (2021)
- [SocAoG: Incremental Graph Parsing for Social Relation Inference in Dialogues](https://mlsanity.com/paper/2106.01006/) (2021)
- [Knowledge Engineering using Large Language Models](https://mlsanity.com/paper/2310.00637/) (2023)
- [Distantly Supervised Relation Extraction with Sentence Reconstruction and Knowledge Base Priors](https://mlsanity.com/paper/2104.08225/) (2021)
- [HiCLRE: A Hierarchical Contrastive Learning Framework for Distantly Supervised Relation Extraction](https://mlsanity.com/paper/2202.13352/) (2022)
- [Why only Micro-F1? Class Weighting of Measures for Relation Classification](https://mlsanity.com/paper/2205.09460/) (2022)
- [Abstractified Multi-instance Learning (AMIL) for Biomedical Relation Extraction](https://mlsanity.com/paper/2110.12501/) (2021)
- [Multilingual Relation Classification via Efficient and Effective Prompting](https://mlsanity.com/paper/2210.13838/) (2022)
- [Completeness, Recall, and Negation in Open-World Knowledge Bases: A Survey](https://mlsanity.com/paper/2305.05403/) (2023)
- [Knowing False Negatives: An Adversarial Training Method for Distantly Supervised Relation Extraction](https://mlsanity.com/paper/2109.02099/) (2021)
- [PARE: A Simple and Strong Baseline for Monolingual and Multilingual Distantly Supervised Relation Extraction](https://mlsanity.com/paper/2110.07415/) (2021)
- [An Overview of Distant Supervision for Relation Extraction with a Focus on Denoising and Pre-training Methods](https://mlsanity.com/paper/2207.08286/) (2022)
- [Sent2Span: Span Detection for PICO Extraction in the Biomedical Text without Span Annotations](https://mlsanity.com/paper/2109.02254/) (2021)
- [MedDistant19: Towards an Accurate Benchmark for Broad-Coverage Biomedical Relation Extraction](https://mlsanity.com/paper/2204.04779/) (2022)
- [Evaluating the Efficacy of Large Language Models in Identifying Phishing Attempts](https://mlsanity.com/paper/2404.15485/) (2024)
- [Two Training Strategies for Improving Relation Extraction over Universal Graph](https://mlsanity.com/paper/2102.06540/) (2021)
- [Cross-stitching Text and Knowledge Graph Encoders for Distantly Supervised Relation Extraction](https://mlsanity.com/paper/2211.01432/) (2022)

## Abstract (arXiv)

Distantly supervised relation extraction is widely used to extract relational facts from text, but suffers from noisy labels. Current relation extraction methods try to alleviate the noise by multi-instance learning and by providing supporting linguistic and contextual information to more efficiently guide the relation classification. While achieving state-of-the-art results, we observed these models to be biased towards recognizing a limited set of relations with high precision, while ignoring those in the long tail. To address this gap, we utilize a pre-trained language model, the OpenAI Generative Pre-trained Transformer (GPT) [Radford et al., 2018]. The GPT and similar models have been shown to capture semantic and syntactic features, and also a notable amount of "common-sense" knowledge, which we hypothesize are important features for recognizing a more diverse set of relations. By extending the GPT to the distantly supervised setting, and fine-tuning it on the NYT10 dataset, we show that it predicts a larger set of distinct relation types with high confidence. Manual and automated evaluation of our model shows that it achieves a state-of-the-art AUC score of 0.422 on the NYT10 dataset, and performs especially well at higher recall levels.