# Cross-stitching Text and Knowledge Graph Encoders for Distantly Supervised Relation Extraction

arXiv: https://arxiv.org/abs/2211.01432 · HTML: https://mlsanity.com/paper/2211.01432/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 21 of its 39 references that are in the map.
- First posted on arXiv 2 Nov 2022.

## Builds on (most cited first, 21 of 21)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Neural Machine Translation by Jointly Learning to Align and Translate](https://mlsanity.com/paper/1409.0473/) (2014)
- [BioBERT: a pre-trained biomedical language representation model for biomedical text mining](https://mlsanity.com/paper/1901.08746/) (2019)
- [Cross-stitch Networks for Multi-task Learning](https://mlsanity.com/paper/1604.03539/) (2016)
- [ScispaCy: Fast and Robust Models for Biomedical Natural Language Processing](https://mlsanity.com/paper/1902.07669/) (2019)
- [Learning from Context or Names? An Empirical Study on Neural Relation Extraction](https://mlsanity.com/paper/2010.01923/) (2020)
- [OpenNRE: An Open and Extensible Toolkit for Neural Relation Extraction](https://mlsanity.com/paper/1909.13078/) (2019)
- [Long-tail Relation Extraction via Knowledge Graph Embeddings and Graph Convolution Networks](https://mlsanity.com/paper/1903.01306/) (2019)
- [RESIDE: Improving Distantly-Supervised Neural Relation Extraction using Side Information](https://mlsanity.com/paper/1812.04361/) (2018)
- [Fine-tuning Pre-Trained Transformer Language Models to Distantly Supervised Relation Extraction](https://mlsanity.com/paper/1906.08646/) (2019)
- [Improving Distantly Supervised Relation Extraction using Word and Entity Based Attention](https://mlsanity.com/paper/1804.06987/) (2018)
- [Distant Supervision Relation Extraction with Intra-Bag and Inter-Bag Attentions](https://mlsanity.com/paper/1904.00143/) (2019)
- [Connecting Language and Knowledge Bases with Embedding Models for Relation Extraction](https://mlsanity.com/paper/1307.7973/) (2013)
- [Self-Attention Enhanced Selective Gate with Entity-Aware Embedding for Distantly Supervised Relation Extraction](https://mlsanity.com/paper/1911.11899/) (2019)
- [Manual Evaluation Matters: Reviewing Test Protocols of Distantly Supervised Relation Extraction](https://mlsanity.com/paper/2105.09543/) (2021)
- [Incorporating Relation Paths in Neural Relation Extraction](https://mlsanity.com/paper/1609.07479/) (2016)
- [Multi-Level Structured Self-Attentions for Distantly Supervised Relation Extraction](https://mlsanity.com/paper/1809.00699/) (2018)
- [How Knowledge Graph and Attention Help? A Quantitative Analysis into Bag-level Relation Extraction](https://mlsanity.com/paper/2107.12064/) (2021)
- [Leveraging 2-hop Distant Supervision from Table Entity Pairs for Relation Extraction](https://mlsanity.com/paper/1909.06007/) (2019)
- [Two Training Strategies for Improving Relation Extraction over Universal Graph](https://mlsanity.com/paper/2102.06540/) (2021)

## Built on by (most cited first, 1 of 1)

- [KnowRA: Knowledge Retrieval Augmented Method for Document-level Relation Extraction with Comprehensive Reasoning Abilities](https://mlsanity.com/paper/2501.00571/) (2024)

## Abstract (arXiv)

Bi-encoder architectures for distantly-supervised relation extraction are designed to make use of the complementary information found in text and knowledge graphs (KG). However, current architectures suffer from two drawbacks. They either do not allow any sharing between the text encoder and the KG encoder at all, or, in case of models with KG-to-text attention, only share information in one direction. Here, we introduce cross-stitch bi-encoders, which allow full interaction between the text encoder and the KG encoder via a cross-stitch mechanism. The cross-stitch mechanism allows sharing and updating representations between the two encoders at any layer, with the amount of sharing being dynamically controlled via cross-attention-based gates. Experimental results on two relation extraction benchmarks from two different domains show that enabling full interaction between the two encoders yields strong improvements.