mlsanityData to 14 Oct 2026

BROS: A Pre-trained Language Model Focusing on Text and Layout for Better Key Information Extraction from Documents

arXiv 2108.04539 ↗First posted 10 Aug 2021cs.CLcited by 104

Not on the map. Why?

Abstract (arXiv)

Key information extraction (KIE) from document images requires understanding the contextual and spatial semantics of texts in two-dimensional (2D) space. Many recent studies try to solve the task by developing pre-trained language models focusing on combining visual features from document images with texts and their layout. On the other hand, this paper tackles the problem by going back to the basic: effective combination of text and layout. Specifically, we propose a pre-trained language model, named BROS (BERT Relying On Spatiality), that encodes relative positions of texts in 2D space and learns from unlabeled documents with area-masking strategy. With this optimized training scheme for understanding texts in 2D space, BROS shows comparable or better performance compared to previous methods on four KIE benchmarks (FUNSD, SROIE*, CORD, and SciTSR) without relying on visual features. This paper also reveals two real-world challenges in KIE tasks-(1) minimizing the error from incorrect text ordering and (2) efficient learning from fewer downstream examples-and demonstrates the superiority of BROS over previous methods. Code is available at https://github.com/clovaai/bros.

Key facts. Cited by 104 papers in the map (data through 14 Oct 2026). Builds on 18 of its 27 references that are in the map. First posted on arXiv 10 Aug 2021.

Builds on

18 of 27 references.

References · 18

  1. 2017Attention Is All You Needcited by 60,361
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  3. 2019Transformer-XL: Attentive Language Models Beyond a Fixed-Length Contextcited by 1,850
  4. 2019SpanBERT: Improving Pre-training by Representing and Predicting Spanscited by 834
  5. 2019LayoutLM: Pre-training of Text and Layout for Document Image Understandingcited by 553
  6. 2020LayoutLMv2: Multi-modal Pre-training for Visually-Rich Document Understandingcited by 408
  7. 2019FUNSD: A Dataset for Form Understanding in Noisy Scanned Documentscited by 349
  8. 2021ICDAR2019 Competition on Scanned Receipt OCR and Information Extractioncited by 256
  9. 2021DocFormer: End-to-End Transformer for Document Understandingcited by 256
  10. 2015Evaluation of Deep Convolutional Nets for Document Image Classification and Retrievalcited by 213
  11. 2021Going Full-TILT Boogie on Document Understanding with Text-Image-Layout Transformercited by 134
  12. 2021SelfDoc: Self-Supervised Document Representation Learningcited by 117
6 more
  1. 2019Complicated Table Structure Recognitioncited by 92
  2. 2021StructuralLM: Structural Pre-training for Form Understandingcited by 91
  3. 2021StrucTexT: Structured Text Understanding with Multi-Modal Transformerscited by 90
  4. 2019Graph Convolution for Multimodal Information Extraction from Visually Rich Documentscited by 88
  5. 2020Spatial Dependency Parsing for Semi-Structured Document Information Extractioncited by 65
  6. 2019BERTgrid: Contextualized Embedding for 2D Document Representation and Understandingcited by 63

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2021: 1; 2022: 17; 2023: 29; 2024: 30; 2025: 19; 2026: 8.

104 papers.

Sort:

By area:

  1. 2022LayoutLMv3: Pre-training for Document AI with Unified Text and Image Maskingcited by 485
  2. 2021OCR-free Document Understanding Transformercited by 407
  3. 2022Unifying Vision, Text, and Layout for Universal Document Processingcited by 140
  4. 2023DocPedia: Unleashing the Power of Large Multimodal Model in the Frequency Domain for Versatile Document Understandingcited by 106
  5. 2024LayoutLLM: Layout Instruction Tuning with Large Language Models for Document Understandingcited by 95
  6. 2024OmniParser: A Unified Framework for Text Spotting, Key Information Extraction and Table Recognitioncited by 74
  7. 2022End-to-end Document Recognition and Understanding with Dessurtcited by 73
  8. 2022XYLayoutLM: Towards Layout-Aware Multimodal Networks For Visually-Rich Document Understandingcited by 68
  9. 2023DocILE Benchmark for Document Information Localization and Extractioncited by 58
  10. 2024A Bounding Box is Worth One Token: Interleaving Layout and Text in a Large Language Model for Document Understandingcited by 57
  11. 2023Exploring OCR Capabilities of GPT-4V(ision) : A Quantitative and In-depth Evaluationcited by 46
  12. 2023GeoLayoutLM: Geometric Pre-training for Visual Information Extractioncited by 40
38 more
  1. 2023ICL-D3IE: In-Context Learning with Diverse Demonstrations Updating for Document Information Extractioncited by 38
  2. 2022Multimodal Pre-training Based on Graph Attention Network for Document Understandingcited by 24
  3. 2023Layout and Task Aware Instruction Prompt for Zero-shot Document Image Question Answeringcited by 24
  4. 2023LANS: A Layout-Aware Neural Solver for Plane Geometry Problemcited by 24
  5. 2023Reading Order Matters: Information Extraction from Visually-rich Documents by Token Path Predictioncited by 23
  6. 2024DocLayLLM: An Efficient Multi-modal Extension of Large Language Models for Text-rich Document Understandingcited by 23
  7. 2025OmniParser V2: Structured-Points-of-Thought for Unified Visual Text Parsing and Its Generality to Multimodal Large Language Modelscited by 23
  8. 2023Modeling Entities as Semantic Points for Visual Information Extraction in the Wildcited by 20
  9. 2023DocParser: End-to-end OCR-free Information Extraction from Visually Rich Documentscited by 19
  10. 2023DocTr: Document Transformer for Structured Information Extraction in Documentscited by 18
  11. 2023Attention Where It Matters: Rethinking Visual Document Understanding with Selective Region Concentrationcited by 18
  12. 2023LayoutMask: Enhance Text-Layout Interaction in Multi-modal Pre-training for Document Understandingcited by 17
  13. 2025MMRAG: Multi-Mode Retrieval-Augmented Generation with Large Language Models for Biomedical In-Context Learningcited by 13
  14. 2022XDoc: Unified Pre-training for Cross-Format Document Understandingcited by 12
  15. 2024Modeling Layout Reading Order as Ordering Relations for Visually-rich Document Understandingcited by 12
  16. 2022Relational Representation Learning in Visually-Rich Documentscited by 11
  17. 2022ERNIE-mmLayout: Multi-grained MultiModal Transformer for Document Understandingcited by 9
  18. 2022MGDoc: Pre-training with Multi-granular Hierarchy for Document Image Understandingcited by 9
  19. 2023A Question-Answering Approach to Key Value Pair Extraction from Form-like Document Imagescited by 9
  20. 2024DANIEL: A fast Document Attention Network for Information Extraction and Labelling of handwritten documentscited by 9
  21. 2024BuDDIE: A Business Document Dataset for Multi-task Information Extractioncited by 8
  22. 2025A Survey on MLLM-based Visually Rich Document Understanding: Methods, Challenges, and Emerging Trendscited by 8
  23. 2022Understanding Long Documents with Different Position-Aware Attentionscited by 7
  24. 2024ReALM: Reference Resolution As Language Modelingcited by 7
  25. 2022GMN: Generative Multi-modal Network for Practical Document Information Extractioncited by 6
  26. 2023Fast-StrucTexT: An Efficient Hourglass Transformer with Modality-guided Dynamic Token Merge for Document Understandingcited by 6
  27. 2023Do-GOOD: Towards Distribution Shift Evaluation for Pre-Trained Visual Document Understanding Modelscited by 6
  28. 2023Enhancing Visually-Rich Document Understanding via Layout Structure Modelingcited by 6
  29. 2024Transformers and Language Models in Form Understanding: A Comprehensive Review of Scanned Document Analysiscited by 6
  30. 2024SAIL: Sample-Centric In-Context Learning for Document Information Extractioncited by 6
  31. 2024XFormParser: A Simple and Effective Multimodal Multilingual Semi-structured Form Parsercited by 5
  32. 2025Document Intelligence in the Era of Large Language Models: A Surveycited by 5
  33. 2023Visually-Situated Natural Language Understanding with Contrastive Reading Model and Frozen Large Language Modelscited by 4
  34. 2023DocumentNet: Bridging the Data Gap in Document Pre-Trainingcited by 4
  35. 2024SynthDoc: Bilingual Documents Synthesis for Visual Document Understandingcited by 4
  36. 2024ReLayout: Towards Real-World Document Understanding via Layout-enhanced Pre-trainingcited by 4
  37. 2025Survey on Question Answering over Visually Rich Documents: Methods, Challenges, and Trendscited by 4
  38. 2023SCOB: Universal Text Understanding via Character-wise Supervised Contrastive Learning with Online Text Rendering for Bridging Domain Gapcited by 3

50 most cited of 104 All 104 (JSON)