# Towards Few-shot Entity Recognition in Document Images: A Graph Neural Network Approach Robust to Image Manipulation

arXiv: https://arxiv.org/abs/2305.14828 · HTML: https://mlsanity.com/paper/2305.14828/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 19 of its 47 references that are in the map.
- First posted on arXiv 24 May 2023.

## Builds on (most cited first, 19 of 19)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [Graph Attention Networks](https://mlsanity.com/paper/1710.10903/) (2017)
- [HuggingFace's Transformers: State-of-the-art Natural Language Processing](https://mlsanity.com/paper/1910.03771/) (2019)
- [Unified Language Model Pre-training for Natural Language Understanding and Generation](https://mlsanity.com/paper/1905.03197/) (2019)
- [LayoutLM: Pre-training of Text and Layout for Document Image Understanding](https://mlsanity.com/paper/1912.13318/) (2019)
- [LayoutLMv3: Pre-training for Document AI with Unified Text and Image Masking](https://mlsanity.com/paper/2204.08387/) (2022)
- [LayoutLMv2: Multi-modal Pre-training for Visually-Rich Document Understanding](https://mlsanity.com/paper/2012.14740/) (2020)
- [FUNSD: A Dataset for Form Understanding in Noisy Scanned Documents](https://mlsanity.com/paper/1905.13538/) (2019)
- [DocFormer: End-to-End Transformer for Document Understanding](https://mlsanity.com/paper/2106.11539/) (2021)
- [Yara Parser: A Fast and Accurate Dependency Parser](https://mlsanity.com/paper/1503.06733/) (2015)
- [Deep Relational Reasoning Graph Network for Arbitrary Shape Text Detection](https://mlsanity.com/paper/2003.07493/) (2020)
- [FormNet: Structural Encoding beyond Sequential Modeling in Form Document Information Extraction](https://mlsanity.com/paper/2203.08411/) (2022)
- [ZeroShotCeres: Zero-Shot Relation Extraction from Semi-Structured Webpages](https://mlsanity.com/paper/2005.07105/) (2020)
- [FormNetV2: Multimodal Graph Contrastive Learning for Form Document Information Extraction](https://mlsanity.com/paper/2305.02549/) (2023)
- [Towards Few-shot Entity Recognition in Document Images: A Label-aware Sequence-to-Sequence Framework](https://mlsanity.com/paper/2204.05819/) (2022)
- [MGDoc: Pre-training with Multi-granular Hierarchy for Document Image Understanding](https://mlsanity.com/paper/2211.14958/) (2022)
- [Domain Agnostic Few-Shot Learning For Document Intelligence](https://mlsanity.com/paper/2111.00007/) (2021)

## Built on by (most cited first, 2 of 2)

- [Modeling Layout Reading Order as Ordering Relations for Visually-rich Document Understanding](https://mlsanity.com/paper/2409.19672/) (2024)
- [Unveiling the Deficiencies of Pre-trained Text-and-Layout Models in Real-world Visually-rich Document Information Extraction](https://mlsanity.com/paper/2402.02379/) (2024)

## Abstract (arXiv)

Recent advances of incorporating layout information, typically bounding box coordinates, into pre-trained language models have achieved significant performance in entity recognition from document images. Using coordinates can easily model the absolute position of each token, but they might be sensitive to manipulations in document images (e.g., shifting, rotation or scaling), especially when the training data is limited in few-shot settings. In this paper, we propose to further introduce the topological adjacency relationship among the tokens, emphasizing their relative position information. Specifically, we consider the tokens in the documents as nodes and formulate the edges based on the topological heuristics from the k-nearest bounding boxes. Such adjacency graphs are invariant to affine transformations including shifting, rotations and scaling. We incorporate these graphs into the pre-trained language model by adding graph neural network layers on top of the language model embeddings, leading to a novel model LAGER. Extensive experiments on two benchmark datasets show that LAGER significantly outperforms strong baselines under different few-shot settings and also demonstrate better robustness to manipulations.