# Modelling the semantics of text in complex document layouts using graph transformer networks

arXiv: https://arxiv.org/abs/2202.09144 · HTML: https://mlsanity.com/paper/2202.09144/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 13 of its 24 references that are in the map.
- First posted on arXiv 18 Feb 2022.

## Builds on (most cited first, 13 of 13)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Graph Attention Networks](https://mlsanity.com/paper/1710.10903/) (2017)
- [Distributed Representations of Words and Phrases and their Compositionality](https://mlsanity.com/paper/1310.4546/) (2013)
- [Quantifying Attention Flow in Transformers](https://mlsanity.com/paper/2005.00928/) (2020)
- [A Generalization of Transformer Networks to Graphs](https://mlsanity.com/paper/2012.09699/) (2020)
- [Deep Graph Contrastive Representation Learning](https://mlsanity.com/paper/2006.04131/) (2020)
- [LayoutLMv2: Multi-modal Pre-training for Visually-Rich Document Understanding](https://mlsanity.com/paper/2012.14740/) (2020)
- [Machine Learning on Graphs: A Model and Comprehensive Taxonomy](https://mlsanity.com/paper/2005.03675/) (2020)
- [StrucTexT: Structured Text Understanding with Multi-Modal Transformers](https://mlsanity.com/paper/2108.02923/) (2021)
- [Representing Numbers in NLP: a Survey and a Vision](https://mlsanity.com/paper/2103.13136/) (2021)
- [TableNet: Deep Learning model for end-to-end Table detection and Tabular data extraction from Scanned Document Images](https://mlsanity.com/paper/2001.01469/) (2020)
- [Evaluation of BERT and ALBERT Sentence Embedding Performance on Downstream NLP Tasks](https://mlsanity.com/paper/2101.10642/) (2021)

## Abstract (arXiv)

Representing structured text from complex documents typically calls for different machine learning techniques, such as language models for paragraphs and convolutional neural networks (CNNs) for table extraction, which prohibits drawing links between text spans from different content types. In this article we propose a model that approximates the human reading pattern of a document and outputs a unique semantic representation for every text span irrespective of the content type they are found in. We base our architecture on a graph representation of the structured text, and we demonstrate that not only can we retrieve semantically similar information across documents but also that the embedding space we generate captures useful semantic information, similar to language models that work only on text sequences.