# Position Masking for Improved Layout-Aware Document Understanding

arXiv: https://arxiv.org/abs/2109.00442 · HTML: https://mlsanity.com/paper/2109.00442/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 14 of its 19 references that are in the map.
- First posted on arXiv 1 Sep 2021.

## Builds on (most cited first, 14 of 14)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Fast R-CNN](https://mlsanity.com/paper/1504.08083/) (2015)
- [Energy and Policy Considerations for Deep Learning in NLP](https://mlsanity.com/paper/1906.02243/) (2019)
- [LayoutLM: Pre-training of Text and Layout for Document Image Understanding](https://mlsanity.com/paper/1912.13318/) (2019)
- [LayoutLMv2: Multi-modal Pre-training for Visually-Rich Document Understanding](https://mlsanity.com/paper/2012.14740/) (2020)
- [FUNSD: A Dataset for Form Understanding in Noisy Scanned Documents](https://mlsanity.com/paper/1905.13538/) (2019)
- [Evaluation of Deep Convolutional Nets for Document Image Classification and Retrieval](https://mlsanity.com/paper/1502.07058/) (2015)
- [Chargrid: Towards Understanding 2D Documents](https://mlsanity.com/paper/1809.08799/) (2018)
- [Graph Convolution for Multimodal Information Extraction from Visually Rich Documents](https://mlsanity.com/paper/1903.11279/) (2019)
- [BERTgrid: Contextualized Embedding for 2D Document Representation and Understanding](https://mlsanity.com/paper/1909.04948/) (2019)
- [What Do Position Embeddings Learn? An Empirical Study of Pre-Trained Language Model Positional Encoding](https://mlsanity.com/paper/2010.04903/) (2020)
- [DocStruct: A Multimodal Method to Extract Hierarchy Structure in Document for General Form Understanding](https://mlsanity.com/paper/2010.11685/) (2020)
- [ZeroShotCeres: Zero-Shot Relation Extraction from Semi-Structured Webpages](https://mlsanity.com/paper/2005.07105/) (2020)
- [VisualWordGrid: Information Extraction From Scanned Documents Using A Multimodal Approach](https://mlsanity.com/paper/2010.02358/) (2020)

## Built on by (most cited first, 3 of 3)

- [GVdoc: Graph-based Visual Document Classification](https://mlsanity.com/paper/2305.17219/) (2023)
- [Information Extraction from Documents: Question Answering vs Token Classification in real-world setups](https://mlsanity.com/paper/2304.10994/) (2023)
- [Diagnostics of cognitive failures in multi-agent expert systems using dynamic evaluation protocols and subsequent mutation of the processing context](https://mlsanity.com/paper/2509.15366/) (2025)

## Abstract (arXiv)

Natural language processing for document scans and PDFs has the potential to enormously improve the efficiency of business processes. Layout-aware word embeddings such as LayoutLM have shown promise for classification of and information extraction from such documents. This paper proposes a new pre-training task called that can improve performance of layout-aware word embeddings that incorporate 2-D position embeddings. We compare models pre-trained with only language masking against models pre-trained with both language masking and position masking, and we find that position masking improves performance by over 5% on a form understanding task.