# Can Language Models Capture Graph Semantics? From Graphs to Language Model and Vice-Versa

arXiv: https://arxiv.org/abs/2206.09259 · HTML: https://mlsanity.com/paper/2206.09259/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 13 of its 30 references that are in the map.
- First posted on arXiv 18 Jun 2022.

## Builds on (most cited first, 13 of 13)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Graph Attention Networks](https://mlsanity.com/paper/1710.10903/) (2017)
- [Efficient Estimation of Word Representations in Vector Space](https://mlsanity.com/paper/1301.3781/) (2013)
- [Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translation](https://mlsanity.com/paper/1406.1078/) (2014)
- [Modeling Relational Data with Graph Convolutional Networks](https://mlsanity.com/paper/1703.06103/) (2017)
- [ConceptNet 5.5: An Open Multilingual Graph of General Knowledge](https://mlsanity.com/paper/1612.03975/) (2016)
- [QA-GNN: Reasoning with Language Models and Knowledge Graphs for Question Answering](https://mlsanity.com/paper/2104.06378/) (2021)
- [Language Models are Open Knowledge Graphs](https://mlsanity.com/paper/2010.11967/) (2020)
- [Knowledge-intensive Language Understanding for Explainable AI](https://mlsanity.com/paper/2108.01174/) (2021)
- [Knowledge Infused Learning (K-IL): Towards Deep Incorporation of Knowledge in Deep Learning](https://mlsanity.com/paper/1912.00512/) (2019)
- [Interpreting Language Models Through Knowledge Graph Extraction](https://mlsanity.com/paper/2111.08546/) (2021)

## Built on by (most cited first, 1 of 1)

- [Towards Automatic Support of Software Model Evolution with Large Language~Models](https://mlsanity.com/paper/2312.12404/) (2023)

## Abstract (arXiv)

Knowledge Graphs are a great resource to capture semantic knowledge in terms of entities and relationships between the entities. However, current deep learning models takes as input distributed representations or vectors. Thus, the graph is compressed in a vectorized representation. We conduct a study to examine if the deep learning model can compress a graph and then output the same graph with most of the semantics intact. Our experiments show that Transformer models are not able to express the full semantics of the input knowledge graph. We find that this is due to the disparity between the directed, relationship and type based information contained in a Knowledge Graph and the fully connected token-token undirected graphical interpretation of the Transformer Attention matrix.