# Enriching language models with graph-based context information to better understand textual data

arXiv: https://arxiv.org/abs/2305.11070 · HTML: https://mlsanity.com/paper/2305.11070/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 14 of its 22 references that are in the map.
- First posted on arXiv 10 May 2023.

## Builds on (most cited first, 14 of 14)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Semi-Supervised Classification with Graph Convolutional Networks](https://mlsanity.com/paper/1609.02907/) (2016)
- [Graph Attention Networks](https://mlsanity.com/paper/1710.10903/) (2017)
- [Knowledge Graphs](https://mlsanity.com/paper/2003.02320/) (2020)
- [Graph Convolutional Networks for Text Classification](https://mlsanity.com/paper/1809.05679/) (2018)
- [KG-BERT: BERT for Knowledge Graph Completion](https://mlsanity.com/paper/1909.03193/) (2019)
- [PyTorch-BigGraph: A Large-scale Graph Embedding System](https://mlsanity.com/paper/1903.12287/) (2019)
- [BertGCN: Transductive Text Classification by Combining GCN and BERT](https://mlsanity.com/paper/2105.05727/) (2021)
- [JAKET: Joint Pre-training of Knowledge Graph and Language Understanding](https://mlsanity.com/paper/2010.00796/) (2020)
- [Characterizing and Detecting Hateful Users on Twitter](https://mlsanity.com/paper/1803.08977/) (2018)
- [A Context-Aware Citation Recommendation Model with BERT and Graph Convolutional Networks](https://mlsanity.com/paper/1903.06464/) (2019)
- [Enriching BERT with Knowledge Graph Embeddings for Document Classification](https://mlsanity.com/paper/1909.08402/) (2019)
- [LG4AV: Combining Language Models and Graph Neural Networks for Author Verification](https://mlsanity.com/paper/2109.01479/) (2021)

## Built on by (most cited first, 1 of 1)

- [When Large Language Models Meet Citation: A Survey](https://mlsanity.com/paper/2309.09727/) (2023)

## Abstract (arXiv)

A considerable number of texts encountered daily are somehow connected with each other. For example, Wikipedia articles refer to other articles via hyperlinks, scientific papers relate to others via citations or (co)authors, while tweets relate via users that follow each other or reshare content. Hence, a graph-like structure can represent existing connections and be seen as capturing the "context" of the texts. The question thus arises if extracting and integrating such context information into a language model might help facilitate a better automated understanding of the text. In this study, we experimentally demonstrate that incorporating graph-based contextualization into BERT model enhances its performance on an example of a classification task. Specifically, on Pubmed dataset, we observed a reduction in error from 8.51% to 7.96%, while increasing the number of parameters just by 1.6%. Our source code: https://github.com/tryptofanik/gc-bert