# LayoutXLM vs. GNN: An Empirical Evaluation of Relation Extraction for Documents

arXiv: https://arxiv.org/abs/2206.10304 · HTML: https://mlsanity.com/paper/2206.10304/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Placed in Document understanding and multimodal RAG, one of 16 directions in Multimodal LLMs / vision-language models.
- Builds on 11 of its 12 references that are in the map.
- First posted on arXiv 9 May 2022.

## Placed in

- Multimodal LLMs / vision-language models / [Document understanding and multimodal RAG](https://mlsanity.com/mllm/document-understanding-and-multimodal-rag/)

## Builds on (most cited first, 11 of 11)

- [Unsupervised Cross-lingual Representation Learning at Scale](https://mlsanity.com/paper/1911.02116/) (2019)
- [LayoutLM: Pre-training of Text and Layout for Document Image Understanding](https://mlsanity.com/paper/1912.13318/) (2019)
- [LayoutLMv2: Multi-modal Pre-training for Visually-Rich Document Understanding](https://mlsanity.com/paper/2012.14740/) (2020)
- [FUNSD: A Dataset for Form Understanding in Noisy Scanned Documents](https://mlsanity.com/paper/1905.13538/) (2019)
- [Residual Gated Graph ConvNets](https://mlsanity.com/paper/1711.07553/) (2017)
- [BROS: A Pre-trained Language Model Focusing on Text and Layout for Better Key Information Extraction from Documents](https://mlsanity.com/paper/2108.04539/) (2021)
- [LayoutXLM: Multimodal Pre-training for Multilingual Visually-rich Document Understanding](https://mlsanity.com/paper/2104.08836/) (2021)
- [Chargrid: Towards Understanding 2D Documents](https://mlsanity.com/paper/1809.08799/) (2018)
- [LAMBERT: Layout-Aware (Language) Modeling for information extraction](https://mlsanity.com/paper/2002.08087/) (2020)
- [Entity Relation Extraction as Dependency Parsing in Visually Rich Documents](https://mlsanity.com/paper/2110.09915/) (2021)
- [Comparing Machine Learning Approaches for Table Recognition in Historical Register Books](https://mlsanity.com/paper/1906.11901/) (2019)

## Built on by (most cited first, 1 of 1)

- [Language Independent Neuro-Symbolic Semantic Parsing for Form Understanding](https://mlsanity.com/paper/2305.04460/) (2023)

## Abstract (arXiv)

This paper investigates the Relation Extraction task in documents by benchmarking two different neural network models: a multi-modal language model (LayoutXLM) and a Graph Neural Network: Edge Convolution Network (ECN). For this benchmark, we use the XFUND dataset, released along with LayoutXLM. While both models reach similar results, they both exhibit very different characteristics. This raises the question on how to integrate various modalities in a neural network: by merging all modalities thanks to additional pretraining (LayoutXLM), or in a cascaded way (ECN). We conclude by discussing some methodological issues that must be considered for new datasets and task definition in the domain of Information Extraction with complex documents.