# BERT-GT: Cross-sentence n-ary relation extraction with BERT and Graph Transformer

arXiv: https://arxiv.org/abs/2101.04158 · HTML: https://mlsanity.com/paper/2101.04158/

## Key facts

- Cited by 10 papers in the map (data through 14 Oct 2026).
- Builds on 13 of its 32 references that are in the map.
- First posted on arXiv 11 Jan 2021.

## Builds on (most cited first, 13 of 13)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [BioBERT: a pre-trained biomedical language representation model for biomedical text mining](https://mlsanity.com/paper/1901.08746/) (2019)
- [Transfer Learning in Biomedical Natural Language Processing: An Evaluation of BERT and ELMo on Ten Benchmarking Datasets](https://mlsanity.com/paper/1906.05474/) (2019)
- [ScispaCy: Fast and Robust Models for Biomedical Natural Language Processing](https://mlsanity.com/paper/1902.07669/) (2019)
- [End-to-End Relation Extraction using LSTMs on Sequences and Tree Structures](https://mlsanity.com/paper/1601.00770/) (2016)
- [Cross-Sentence N-ary Relation Extraction with Graph LSTMs](https://mlsanity.com/paper/1708.03743/) (2017)
- [Attention Guided Graph Convolutional Networks for Relation Extraction](https://mlsanity.com/paper/1906.07510/) (2019)
- [Graph Transformer for Graph-to-Sequence Learning](https://mlsanity.com/paper/1911.07470/) (2019)
- [Distant Supervision for Relation Extraction beyond the Sentence Boundary](https://mlsanity.com/paper/1609.04873/) (2016)
- [Simultaneously Self-Attending to All Mentions for Full-Abstract Biological Relation Extraction](https://mlsanity.com/paper/1802.10569/) (2018)
- [N-ary Relation Extraction using Graph State LSTM](https://mlsanity.com/paper/1808.09101/) (2018)
- [Semantic Relation Classification via Convolutional Neural Networks with Simple Negative Sampling](https://mlsanity.com/paper/1506.07650/) (2015)

## Built on by (most cited first, 10 of 10)

- [BioRED: A Rich Biomedical Relation Extraction Dataset](https://mlsanity.com/paper/2204.04263/) (2022)
- [A sequence-to-sequence approach for document-level relation extraction](https://mlsanity.com/paper/2204.01098/) (2022)
- [Transformers in Healthcare: A Survey](https://mlsanity.com/paper/2307.00067/) (2023)
- [A Comprehensive Survey of Document-level Relation Extraction (2016-2023)](https://mlsanity.com/paper/2309.16396/) (2023)
- [Multi-Attribute Relation Extraction (MARE) -- Simplifying the Application of Relation Extraction](https://mlsanity.com/paper/2111.09035/) (2021)
- [High-throughput Biomedical Relation Extraction for Semi-Structured Web Articles Empowered by Large Language Models](https://mlsanity.com/paper/2312.08274/) (2023)
- [Merged ChemProt-DrugProt for Relation Extraction from Biomedical Literature](https://mlsanity.com/paper/2405.18605/) (2024)
- [Biomedical Relation Extraction via Adaptive Document-Relation Cross-Mapping and Concept Unique Identifier](https://mlsanity.com/paper/2501.05155/) (2025)
- [Enhancing Biomedical Relation Extraction with Directionality](https://mlsanity.com/paper/2501.14079/) (2025)
- [EMBRE: Entity-aware Masking for Biomedical Relation Extraction](https://mlsanity.com/paper/2401.07877/) (2024)

## Abstract (arXiv)

A biomedical relation statement is commonly expressed in multiple sentences and consists of many concepts, including gene, disease, chemical, and mutation. To automatically extract information from biomedical literature, existing biomedical text-mining approaches typically formulate the problem as a cross-sentence n-ary relation-extraction task that detects relations among n entities across multiple sentences, and use either a graph neural network (GNN) with long short-term memory (LSTM) or an attention mechanism. Recently, Transformer has been shown to outperform LSTM on many natural language processing (NLP) tasks. In this work, we propose a novel architecture that combines Bidirectional Encoder Representations from Transformers with Graph Transformer (BERT-GT), through integrating a neighbor-attention mechanism into the BERT architecture. Unlike the original Transformer architecture, which utilizes the whole sentence(s) to calculate the attention of the current token, the neighbor-attention mechanism in our method calculates its attention utilizing only its neighbor tokens. Thus, each token can pay attention to its neighbor information with little noise. We show that this is critically important when the text is very long, as in cross-sentence or abstract-level relation-extraction tasks. Our benchmarking results show improvements of 5.44% and 3.89% in accuracy and F1-measure over the state-of-the-art on n-ary and chemical-protein relation datasets, suggesting BERT-GT is a robust approach that is applicable to other biomedical relation extraction tasks or datasets.