# Deep Bidirectional Transformers for Relation Extraction without Supervision

arXiv: https://arxiv.org/abs/1911.00313 · HTML: https://mlsanity.com/paper/1911.00313/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 1 Nov 2019.

## Built on by (most cited first, 5 of 5)

- [Discourse Probing of Pretrained Language Models](https://mlsanity.com/paper/2104.05882/) (2021)
- [What Can Knowledge Bring to Machine Learning? -- A Survey of Low-shot Learning for Structured Data](https://mlsanity.com/paper/2106.06410/) (2021)
- [SocAoG: Incremental Graph Parsing for Social Relation Inference in Dialogues](https://mlsanity.com/paper/2106.01006/) (2021)
- [Towards Understanding Large-Scale Discourse Structures in Pre-Trained and Fine-Tuned Language Models](https://mlsanity.com/paper/2204.04289/) (2022)
- [EDeR: A Dataset for Exploring Dependency Relations Between Events](https://mlsanity.com/paper/2304.01612/) (2023)

## Abstract (arXiv)

We present a novel framework to deal with relation extraction tasks in cases where there is complete lack of supervision, either in the form of gold annotations, or relations from a knowledge base. Our approach leverages syntactic parsing and pre-trained word embeddings to extract few but precise relations,which are then used to annotate a larger cor-pus, in a manner identical to distant supervision. The resulting data set is employed to fine tune a pre-trained BERT model in order to perform relation extraction. Empirical evaluation on four data sets from the biomedical domain shows that our method significantly outperforms two simple baselines for unsupervised relation extraction and, even if not using any supervision at all, achieves slightly worse results than the state-of-the-art in three out of four data sets. Importantly, we show that it is possible to successfully fine tune a large pre-trained language model with noisy data, as op-posed to previous works that rely on gold data for fine tuning.