# From Consensus to Disagreement: Multi-Teacher Distillation for Semi-Supervised Relation Extraction

arXiv: https://arxiv.org/abs/2112.01048 · HTML: https://mlsanity.com/paper/2112.01048/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 22 of its 44 references that are in the map.
- First posted on arXiv 2 Dec 2021.

## Builds on (most cited first, 22 of 22)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Distilling the Knowledge in a Neural Network](https://mlsanity.com/paper/1503.02531/) (2015)
- [Visualizing the Loss Landscape of Neural Nets](https://mlsanity.com/paper/1712.09913/) (2017)
- [Deep Mutual Learning](https://mlsanity.com/paper/1706.00384/) (2017)
- [Improved Knowledge Distillation via Teacher Assistant](https://mlsanity.com/paper/1902.03393/) (2019)
- [Do Deep Nets Really Need to be Deep?](https://mlsanity.com/paper/1312.6184/) (2013)
- [LUKE: Deep Contextualized Entity Representations with Entity-aware Self-attention](https://mlsanity.com/paper/2010.01057/) (2020)
- [Matching the Blanks: Distributional Similarity for Relation Learning](https://mlsanity.com/paper/1906.03158/) (2019)
- [BAM! Born-Again Multi-Task Networks for Natural Language Understanding](https://mlsanity.com/paper/1907.04829/) (2019)
- [Attention Guided Graph Convolutional Networks for Relation Extraction](https://mlsanity.com/paper/1906.07510/) (2019)
- [Learning from Context or Names? An Empirical Study on Neural Relation Extraction](https://mlsanity.com/paper/2010.01923/) (2020)
- [Relation Extraction : A Survey](https://mlsanity.com/paper/1712.05191/) (2017)
- [Semi-supervised Relation Extraction via Incremental Meta Self-Training](https://mlsanity.com/paper/2010.16410/) (2020)
- [LRC-BERT: Latent-representation Contrastive Knowledge Distillation for Natural Language Understanding](https://mlsanity.com/paper/2012.07335/) (2020)
- [NERO: A Neural Rule Grounding Framework for Label-Efficient Relation Extraction](https://mlsanity.com/paper/1909.02177/) (2019)
- [Topology Distillation for Recommender System](https://mlsanity.com/paper/2106.08700/) (2021)
- [Probing Linguistic Features of Sentence-Level Representations in Neural Relation Extraction](https://mlsanity.com/paper/2004.08134/) (2020)
- [Learning Dual Retrieval Module for Semi-supervised Relation Extraction](https://mlsanity.com/paper/1902.07814/) (2019)
- [Distilling Structured Knowledge into Embeddings for Explainable and Accurate Recommendation](https://mlsanity.com/paper/1912.08422/) (2019)
- [Incorporating Relation Paths in Neural Relation Extraction](https://mlsanity.com/paper/1609.07479/) (2016)
- [Empower Distantly Supervised Relation Extraction with Collaborative Adversarial Training](https://mlsanity.com/paper/2106.10835/) (2021)
- [Looking Beyond Label Noise: Shifted Label Distribution Matters in Distantly Supervised Relation Extraction](https://mlsanity.com/paper/1904.09331/) (2019)

## Abstract (arXiv)

Lack of labeled data is a main obstacle in relation extraction. Semi-supervised relation extraction (SSRE) has been proven to be a promising way for this problem through annotating unlabeled samples as additional training data. Almost all prior researches along this line adopt multiple models to make the annotations more reliable by taking the intersection set of predicted results from these models. However, the difference set, which contains rich information about unlabeled data, has been long neglected by prior studies. In this paper, we propose to learn not only from the consensus but also the disagreement among different models in SSRE. To this end, we develop a simple and general multi-teacher distillation (MTD) framework, which can be easily integrated into any existing SSRE methods. Specifically, we first let the teachers correspond to the multiple models and select the samples in the intersection set of the last iteration in SSRE methods to augment labeled data as usual. We then transfer the class distributions for samples in the difference set as soft labels to guide the student. We finally perform prediction using the trained student model. Experimental results on two public datasets demonstrate that our framework significantly promotes the performance of the base SSRE methods with pretty low computational cost.