mlsanityData to 14 Oct 2026

MarkupLM: Pre-training of Text and Markup Language for Visually-rich Document Understanding

arXiv 2110.08518 ↗First posted 16 Oct 2021cs.CLcited by 45

Not on the map. Why?

Abstract (arXiv)

Multimodal pre-training with text, layout, and image has made significant progress for Visually Rich Document Understanding (VRDU), especially the fixed-layout documents such as scanned document images. While, there are still a large number of digital documents where the layout information is not fixed and needs to be interactively and dynamically rendered for visualization, making existing layout-based pre-training approaches not easy to apply. In this paper, we propose MarkupLM for document understanding tasks with markup languages as the backbone, such as HTML/XML-based documents, where text and markup information is jointly pre-trained. Experiment results show that the pre-trained MarkupLM significantly outperforms the existing strong baseline models on several document understanding tasks. The pre-trained model and code will be publicly available at https://aka.ms/markuplm.

Key facts. Cited by 45 papers in the map (data through 14 Oct 2026). Builds on 25 of its 34 references that are in the map. First posted on arXiv 16 Oct 2021.

Builds on

25 of 34 references.

References · 25

  1. 2017Attention Is All You Needcited by 60,361
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  3. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,372
  4. 2015Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networkscited by 9,639
  5. 2016SQuAD: 100,000+ Questions for Machine Comprehension of Textcited by 4,148
  6. 2020DocVQA: A Dataset for VQA on Document Imagescited by 1,317
  7. 2016Bag of Tricks for Efficient Text Classificationcited by 876
  8. 2016Training Deep Nets with Sublinear Memory Costcited by 848
  9. 2019LayoutLM: Pre-training of Text and Layout for Document Image Understandingcited by 553
  10. 2020LayoutLMv2: Multi-modal Pre-training for Visually-Rich Document Understandingcited by 408
  11. 2019FUNSD: A Dataset for Form Understanding in Noisy Scanned Documentscited by 349
  12. 2021ICDAR2019 Competition on Scanned Receipt OCR and Information Extractioncited by 256
13 more
  1. 2021DocFormer: End-to-End Transformer for Document Understandingcited by 256
  2. 2015Evaluation of Deep Convolutional Nets for Document Image Classification and Retrievalcited by 213
  3. 2021Going Full-TILT Boogie on Document Understanding with Text-Image-Layout Transformercited by 134
  4. 2021SelfDoc: Self-Supervised Document Representation Learningcited by 117
  5. 2021LayoutXLM: Multimodal Pre-training for Multilingual Visually-rich Document Understandingcited by 103
  6. 2021WebSRC: A Dataset for Web-Based Structural Reading Comprehensioncited by 100
  7. 2021StructuralLM: Structural Pre-training for Form Understandingcited by 91
  8. 2020LAMBERT: Layout-Aware (Language) Modeling for information extractioncited by 53
  9. 2020Kleister: A novel task for Information Extraction involving Long Documents with Complex Layoutcited by 27
  10. 2020FreeDOM: A Transferable Neural Architecture for Structured Information Extraction on Web Documentscited by 22
  11. 2020Towards a Multi-modal, Multi-task Learning based Pre-training Framework for Document Representation Learningcited by 21
  12. 2021Simplified DOM Trees for Transferable Attribute Extraction from the Webcited by 16
  13. 2021LAMPRET: Layout-Aware Multimodal PreTraining for Document Understandingcited by 15

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2021: 1; 2022: 11; 2023: 9; 2024: 13; 2025: 4; 2026: 7.

45 papers.

Sort:

By area:

  1. 2022Pix2Struct: Screenshot Parsing as Pretraining for Visual Language Understandingcited by 366
  2. 2023A Real-World WebAgent with Planning, Long Context Understanding, and Program Synthesiscited by 353
  3. 2024WebLINX: Real-World Website Navigation with Multi-Turn Dialoguecited by 180
  4. 2023Multimodal Web Navigation with Instruction-Finetuned Foundation Modelscited by 160
  5. 2023KOSMOS-2.5: A Multimodal Literate Modelcited by 96
  6. 2021Document AI: Benchmarks, Models and Applicationscited by 62
  7. 2023DocFormerv2: Local Features for Document Understandingcited by 46
  8. 2022Table Pre-training: A Survey on Model Architectures, Pre-training Objectives, and Downstream Taskscited by 42
  9. 2023Schema-Driven Information Extraction from Heterogeneous Tablescited by 25
  10. 2022DOM-LM: Learning Generalizable Representations for HTML Documentscited by 24
  11. 2024Deep Learning based Visually Rich Document Content Understanding: A Surveycited by 21
  12. 2024AutoScraper: A Progressive Understanding Web Agent for Web Scraper Generationcited by 20
33 more
  1. 2025LLM/Agent-as-Data-Analyst: A Surveycited by 16
  2. 2024Enhancing Vision-Language Pre-training with Rich Supervisionscited by 13
  3. 2022XDoc: Unified Pre-training for Cross-Format Document Understandingcited by 12
  4. 2024LAPDoc: Layout-Aware Prompting for Documentscited by 9
  5. 2022Do BERTs Learn to Use Browser User Interface? Exploring Multi-Step Tasks with Unified Vision-and-Language BERTscited by 5
  6. 2022On Web-based Visual Corpus Construction for Visual Document Understandingcited by 5
  7. 2024Cleaner Pretraining Corpus Curation with Neural Web Scrapingcited by 5
  8. 2022TIE: Topological Information Enhanced Structural Reading Comprehension on Web Pagescited by 4
  9. 2024Hierarchical Multimodal Pre-training for Visually Rich Webpage Understandingcited by 4
  10. 2024WebRPG: Automatic Web Rendering Parameters Generation for Visual Presentationcited by 4
  11. 2022PLAtE: A Large-scale Dataset for List Page Web Extractioncited by 3
  12. 2022FormLM: Recommending Creation Ideas for Online Forms by Modelling Semantic and Structural Informationcited by 3
  13. 2024Enhancing Mobile "How-to" Queries with Automated Search Results Verification and Rerankingcited by 3
  14. 2023Multimodal Document Analytics for Banking Process Automationcited by 2
  15. 2024XPath Agent: An Efficient XPath Programming Agent Based on LLM for Web Crawlercited by 2
  16. 2025NEXT-EVAL: Next Evaluation of Traditional and LLM Web Data Record Extractioncited by 2
  17. 2022GROWN+UP: A Graph Representation Of a Webpage Network Utilizing Pre-trainingcited by 1
  18. 2023Towards Zero-shot Relation Extraction in Web Mining: A Multimodal Approach with Relative XML Pathcited by 1
  19. 2024PAFFA: Premeditated Actions For Fast Agentscited by 1
  20. 2025Multilingual Attribute Extraction from News Web Pagescited by 1
  21. 2026AXE: Low-Cost Cross-Domain Web Structured Information Extractioncited by 1
  22. 2022An Inclusive Notion of Textcited by 0
  23. 2023PLM-GNN: A Webpage Classification Method based on Joint Pre-trained Language Model and Graph Neural Networkcited by 0
  24. 2023URL-BERT: Training Webpage Representations via Social Media Engagementscited by 0
  25. 2024AutoFAIR : Automatic Data FAIRification via Machine Readingcited by 0
  26. 2024A comprehensive study of on-device NLP applications -- VQA, automated Form filling, Smart Replies for Linguistic Codeswitchingcited by 0
  27. 2025Cross-Domain Web Information Extraction at Pinterestcited by 0
  28. 2026.tmu: A Low-Entropy Tree-Structured Representation for LLM-Assisted Scientific Writingcited by 0
  29. 2026LiveWeb-IE: A Benchmark For Online Web Information Extractioncited by 0
  30. 2026ReFinE: Streamlining UI Mockup Iteration with Research Findingscited by 0
  31. 2026REGREACT: Self-Correcting Multi-Agent Pipelines for Structured Regulatory Information Extractioncited by 0
  32. 2026COHERENCE: Benchmarking Fine-Grained Image-Text Alignment in Interleaved Multimodal Contextscited by 0
  33. 2026Co-Scraper: query-aware DOM Pruning and Reusable Scraper Synthesis for Lightweight Web Data Extractioncited by 0