# Are Layout-Infused Language Models Robust to Layout Distribution Shifts? A Case Study with Scientific Documents

arXiv: https://arxiv.org/abs/2306.01058 · HTML: https://mlsanity.com/paper/2306.01058/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 20 of its 26 references that are in the map.
- First posted on arXiv 1 Jun 2023.

## Builds on (most cited first, 20 of 20)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [Shortcut Learning in Deep Neural Networks](https://mlsanity.com/paper/2004.07780/) (2020)
- [SciBERT: A Pretrained Language Model for Scientific Text](https://mlsanity.com/paper/1903.10676/) (2019)
- [WILDS: A Benchmark of in-the-Wild Distribution Shifts](https://mlsanity.com/paper/2012.07421/) (2020)
- [LayoutLM: Pre-training of Text and Layout for Document Image Understanding](https://mlsanity.com/paper/1912.13318/) (2019)
- [LayoutLMv3: Pre-training for Document AI with Unified Text and Image Masking](https://mlsanity.com/paper/2204.08387/) (2022)
- [LayoutLMv2: Multi-modal Pre-training for Visually-Rich Document Understanding](https://mlsanity.com/paper/2012.14740/) (2020)
- [OCR-free Document Understanding Transformer](https://mlsanity.com/paper/2111.15664/) (2021)
- [PubLayNet: largest dataset ever for document layout analysis](https://mlsanity.com/paper/1908.07836/) (2019)
- [FUNSD: A Dataset for Form Understanding in Noisy Scanned Documents](https://mlsanity.com/paper/1905.13538/) (2019)
- [DocBank: A Benchmark Dataset for Document Layout Analysis](https://mlsanity.com/paper/2006.01038/) (2020)
- [CrossFit: A Few-shot Learning Challenge for Cross-task Generalization in NLP](https://mlsanity.com/paper/2104.08835/) (2021)
- [SelfDoc: Self-Supervised Document Representation Learning](https://mlsanity.com/paper/2106.03331/) (2021)
- [The Effect of Natural Distribution Shift on Question Answering Models](https://mlsanity.com/paper/2004.14444/) (2020)
- [ERNIE-Layout: Layout Knowledge Enhanced Pre-training for Visually-rich Document Understanding](https://mlsanity.com/paper/2210.06155/) (2022)
- [Kleister: A novel task for Information Extraction involving Long Documents with Complex Layout](https://mlsanity.com/paper/2003.02356/) (2020)
- [VILA: Improving Structured Content Extraction from Scientific PDFs Using Visual Layout Groups](https://mlsanity.com/paper/2106.00676/) (2021)
- [Pre-train or Annotate? Domain Adaptation with a Constrained Budget](https://mlsanity.com/paper/2109.04711/) (2021)
- [LoRaLay: A Multilingual and Multimodal Dataset for Long Range and Layout-Aware Summarization](https://mlsanity.com/paper/2301.11312/) (2023)

## Built on by (most cited first, 4 of 4)

- [The Next Phase of Scientific Fact-Checking: Advanced Evidence Retrieval from Complex Structured Academic Papers](https://mlsanity.com/paper/2506.20844/) (2025)
- [Beyond Human Annotation: Recent Advances in Data Generation Methods for Document Intelligence](https://mlsanity.com/paper/2601.12318/) (2026)
- [A Multi-Modal Multilingual Benchmark for Document Image Classification](https://mlsanity.com/paper/2310.16356/) (2023)
- [CRAWLDoc: A Dataset for Robust Ranking of Bibliographic Documents](https://mlsanity.com/paper/2506.03822/) (2025)

## Abstract (arXiv)

Recent work has shown that infusing layout features into language models (LMs) improves processing of visually-rich documents such as scientific papers. Layout-infused LMs are often evaluated on documents with familiar layout features (e.g., papers from the same publisher), but in practice models encounter documents with unfamiliar distributions of layout features, such as new combinations of text sizes and styles, or new spatial configurations of textual elements. In this work we test whether layout-infused LMs are robust to layout distribution shifts. As a case study we use the task of scientific document structure recovery, segmenting a scientific paper into its structural categories (e.g., "title", "caption", "reference"). To emulate distribution shifts that occur in practice we re-partition the GROTOAP2 dataset. We find that under layout distribution shifts model performance degrades by up to 20 F1. Simple training strategies, such as increasing training diversity, can reduce this degradation by over 35% relative F1; however, models fail to reach in-distribution performance in any tested out-of-distribution conditions. This work highlights the need to consider layout distribution shifts during model evaluation, and presents a methodology for conducting such evaluations.