# Information Extraction from Visually Rich Documents using LLM-based Organization of Documents into Independent Textual Segments arXiv: https://arxiv.org/abs/2505.13535 ยท HTML: https://mlsanity.com/paper/2505.13535/ ## Key facts - Cited by 9 papers in the map (data through 14 Oct 2026). - Builds on 34 of its 51 references that are in the map. - First posted on arXiv 18 May 2025. ## Builds on (most cited first, 30 of 34) - [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018) - [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020) - [LoRA: Low-Rank Adaptation of Large Language Models](https://mlsanity.com/paper/2106.09685/) (2021) - [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019) - [Qwen2.5 Technical Report](https://mlsanity.com/paper/2412.15115/) (2024) - [Improved Baselines with Visual Instruction Tuning](https://mlsanity.com/paper/2310.03744/) (2023) - [DocVQA: A Dataset for VQA on Document Images](https://mlsanity.com/paper/2007.00398/) (2020) - [LayoutLM: Pre-training of Text and Layout for Document Image Understanding](https://mlsanity.com/paper/1912.13318/) (2019) - [LayoutLMv3: Pre-training for Document AI with Unified Text and Image Masking](https://mlsanity.com/paper/2204.08387/) (2022) - [LayoutLMv2: Multi-modal Pre-training for Visually-Rich Document Understanding](https://mlsanity.com/paper/2012.14740/) (2020) - [FUNSD: A Dataset for Form Understanding in Noisy Scanned Documents](https://mlsanity.com/paper/1905.13538/) (2019) - [ICDAR2019 Competition on Scanned Receipt OCR and Information Extraction](https://mlsanity.com/paper/2103.10213/) (2021) - [DocFormer: End-to-End Transformer for Document Understanding](https://mlsanity.com/paper/2106.11539/) (2021) - [Evaluation of Deep Convolutional Nets for Document Image Classification and Retrieval](https://mlsanity.com/paper/1502.07058/) (2015) - [UniLMv2: Pseudo-Masked Language Models for Unified Language Model Pre-Training](https://mlsanity.com/paper/2002.12804/) (2020) - [Going Full-TILT Boogie on Document Understanding with Text-Image-Layout Transformer](https://mlsanity.com/paper/2102.09550/) (2021) - [A Systematic Study and Comprehensive Evaluation of ChatGPT on Benchmark Datasets](https://mlsanity.com/paper/2305.18486/) (2023) - [DocLLM: A layout-aware generative language model for multimodal document understanding](https://mlsanity.com/paper/2401.00908/) (2023) - [Chargrid: Towards Understanding 2D Documents](https://mlsanity.com/paper/1809.08799/) (2018) - [LayoutLLM: Layout Instruction Tuning with Large Language Models for Document Understanding](https://mlsanity.com/paper/2404.05225/) (2024) - [ERNIE-Layout: Layout Knowledge Enhanced Pre-training for Visually-rich Document Understanding](https://mlsanity.com/paper/2210.06155/) (2022) - [Spatial Dependency Parsing for Semi-Structured Document Information Extraction](https://mlsanity.com/paper/2005.00642/) (2020) - [BERTgrid: Contextualized Embedding for 2D Document Representation and Understanding](https://mlsanity.com/paper/1909.04948/) (2019) - [GeoLayoutLM: Geometric Pre-training for Visual Information Extraction](https://mlsanity.com/paper/2304.10759/) (2023) - [LMDX: Language Model-based Document Information Extraction and Localization](https://mlsanity.com/paper/2309.10952/) (2023) - [CloudScan - A configuration-free invoice analysis system using recurrent neural networks](https://mlsanity.com/paper/1708.07403/) (2017) - [Wukong-Reader: Multi-modal Pre-training for Fine-grained Visual Document Understanding](https://mlsanity.com/paper/2212.09621/) (2022) - [Modeling Entities as Semantic Points for Visual Information Extraction in the Wild](https://mlsanity.com/paper/2303.13095/) (2023) - [FormNetV2: Multimodal Graph Contrastive Learning for Form Document Information Extraction](https://mlsanity.com/paper/2305.02549/) (2023) - [Recent Advances in Named Entity Recognition: A Comprehensive Survey and Comparative Study](https://mlsanity.com/paper/2401.10825/) (2024) ## Built on by (most cited first, 9 of 9) - [LAD-RAG: Layout-aware Dynamic RAG for Visually-Rich Document Understanding](https://mlsanity.com/paper/2510.07233/) (2025) - [MedStruct-S: A Benchmark for Key Discovery, Key-Conditioned QA and Semi-Structured Extraction from OCR Clinical Reports](https://mlsanity.com/paper/2605.03103/) (2026) - [CTI2TTP: Evidence-Grounded TTP Extraction from Cyber Threat Intelligence via Diverge-then-Converge Verification](https://mlsanity.com/paper/2605.25836/) (2026) - [Enhancing Large Multimodal Models in Key Information Extraction via Scene-Aware Document Synthesis](https://mlsanity.com/paper/2607.04636/) (2026) - [SAGE: Governed Artifact Generation from Enterprise Guidelines](https://mlsanity.com/paper/2609.17775/) (2026) - [Multimodal Approaches for Visually-Rich Document Type Classification: A Comparative Analysis](https://mlsanity.com/paper/2606.02162/) (2026) - [Task Decomposition for Efficient Annotation](https://mlsanity.com/paper/2606.24734/) (2026) - [LAVA: Logic-Aware Validation and Augmentation Framework for Large-Scale Financial Document Auditing](https://mlsanity.com/paper/2608.16763/) (2026) - [DocMIDE: Learning Multi-Hop Implicit Derivation in Visually Rich Documents](https://mlsanity.com/paper/2609.24092/) (2026) ## Abstract (arXiv) Information extraction (IE) from Visually Rich Documents (VRDs) containing layout features along with text is a critical and well-studied task. Specialized non-LLM NLP-based solutions typically involve training models using both textual and geometric information to label sequences/tokens as named entities or answers to specific questions. However, these approaches lack reasoning, are not able to infer values not explicitly present in documents, and do not generalize well to new formats. Generative LLM-based approaches proposed recently are capable of reasoning, but struggle to comprehend clues from document layout especially in previously unseen document formats, and do not show competitive performance in heterogeneous VRD benchmark datasets. In this paper, we propose BLOCKIE, a novel LLM-based approach that organizes VRDs into localized, reusable semantic textual segments called textitsemantic blocks, which are processed independently. Through focused and more generalizable reasoning,our approach outperforms the state-of-the-art on public VRD benchmarks by 1-3% in F1 scores, is resilient to document formats previously not encountered and shows abilities to correctly extract information not explicitly present in documents.