mlsanityData to 14 Oct 2026

DiT: Self-supervised Pre-training for Document Image Transformer

arXiv 2203.02378 ↗First posted 4 Mar 2022cs.CVcited by 152

Not on the map. Why?

Abstract (arXiv)

Image Transformer has recently achieved significant progress for natural image understanding, either using supervised (ViT, DeiT, etc.) or self-supervised (BEiT, MAE, etc.) pre-training techniques. In this paper, we propose DiT, a self-supervised pre-trained Document Image Transformer model using large-scale unlabeled text images for Document AI tasks, which is essential since no supervised counterparts ever exist due to the lack of human-labeled document images. We leverage DiT as the backbone network in a variety of vision-based Document AI tasks, including document image classification, document layout analysis, table detection as well as text detection for OCR. Experiment results have illustrated that the self-supervised pre-trained DiT model achieves new state-of-the-art results on these downstream tasks, e.g. document image classification (91.11 rightarrow 92.69), document layout analysis (91.0 rightarrow 94.9), table detection (94.23 rightarrow 96.55) and text detection for OCR (93.07 rightarrow 94.29). The code and pre-trained models are publicly available at urlhttps://aka.ms/msdit.

Key facts. Cited by 152 papers in the map (data through 14 Oct 2026). Builds on 38 of its 50 references that are in the map. First posted on arXiv 4 Mar 2022.

Builds on

38 of 50 references.

References · 38

  1. 2017Attention Is All You Needcited by 60,361
  2. 2014Adam: A Method for Stochastic Optimizationcited by 37,240
  3. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  4. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,139
  5. 2020End-to-End Object Detection with Transformerscited by 7,414
  6. 2021Masked Autoencoders Are Scalable Vision Learnerscited by 7,403
  7. 2017Mask R-CNNcited by 7,277
  8. 2021Emerging Properties in Self-Supervised Vision Transformerscited by 6,968
  9. 2020Training data-efficient image transformers & distillation through attentioncited by 4,586
  10. 2021Zero-Shot Text-to-Image Generationcited by 4,288
  11. 2016Aggregated Residual Transformations for Deep Neural Networkscited by 2,546
  12. 2021BEiT: BERT Pre-Training of Image Transformerscited by 2,375
26 more
  1. 2021An Empirical Study of Training Self-Supervised Vision Transformerscited by 1,616
  2. 2017Cascade R-CNN: Delving into High Quality Object Detectioncited by 1,276
  3. 2016Deep Networks with Stochastic Depthcited by 979
  4. 2021iBOT: Image BERT Pre-Training with Online Tokenizercited by 903
  5. 2019LayoutLM: Pre-training of Text and Layout for Document Image Understandingcited by 553
  6. 2020LayoutLMv2: Multi-modal Pre-training for Visually-Rich Document Understandingcited by 408
  7. 2019PubLayNet: largest dataset ever for document layout analysiscited by 351
  8. 2019FUNSD: A Dataset for Form Understanding in Noisy Scanned Documentscited by 349
  9. 2021XCiT: Cross-Covariance Image Transformerscited by 338
  10. 2021DocFormer: End-to-End Transformer for Document Understandingcited by 256
  11. 2019Image-based table recognition: data, model, and evaluationcited by 237
  12. 2019Real-time Scene Text Detection with Differentiable Binarizationcited by 214
  13. 2015Evaluation of Deep Convolutional Nets for Document Image Classification and Retrievalcited by 213
  14. 2020DocBank: A Benchmark Dataset for Document Layout Analysiscited by 153
  15. 2021Going Full-TILT Boogie on Document Understanding with Text-Image-Layout Transformercited by 134
  16. 2021Benchmarking Detection Transfer Learning with Vision Transformerscited by 131
  17. 2021SelfDoc: Self-Supervised Document Representation Learningcited by 117
  18. 2021LayoutXLM: Multimodal Pre-training for Multilingual Visually-rich Document Understandingcited by 103
  19. 2019TableBank: A Benchmark Dataset for Table Detection and Recognitioncited by 102
  20. 2021StructuralLM: Structural Pre-training for Form Understandingcited by 91
  21. 2021Document AI: Benchmarks, Models and Applicationscited by 62
  22. 2020LAMBERT: Layout-Aware (Language) Modeling for information extractioncited by 53
  23. 2017Cutting the Error by Half: Investigation of Very Deep CNN and Advanced Training Strategies for Document Image Classificationcited by 22
  24. 2020Towards a Multi-modal, Multi-task Learning based Pre-training Framework for Document Representation Learningcited by 21
  25. 2018Document Image Classification with Intra-Domain Transfer Learning and Stacked Generalization of Deep Convolutional Neural Networkscited by 20
  26. 2021LAMPRET: Layout-Aware Multimodal PreTraining for Document Understandingcited by 15

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2022: 16; 2023: 43; 2024: 36; 2025: 32; 2026: 25.

152 papers.

Sort:

By area:

  1. 2022LayoutLMv3: Pre-training for Document AI with Unified Text and Image Maskingcited by 485
  2. 2022Multimodal Learning with Transformers: A Surveycited by 400
  3. 2023LLaVAR: Enhanced Visual Instruction Tuning for Text-Rich Image Understandingcited by 274
  4. 2024OmniDocBench: Benchmarking Diverse PDF Document Parsing with Comprehensive Annotationscited by 150
  5. 2023Document Understanding Dataset and Evaluation (DUDE)cited by 135
  6. 2023DocLLM: A layout-aware generative language model for multimodal document understandingcited by 116
  7. 2022Hierarchical multimodal transformers for Multi-Page DocVQAcited by 105
  8. 2023KOSMOS-2.5: A Multimodal Literate Modelcited by 96
  9. 2024DeMamba: AI-Generated Video Detection on Million-Scale GenVideo Benchmarkcited by 92
  10. 2023Expressive Text-to-Image Generation with Rich Textcited by 89
  11. 2024DocLayout-YOLO: Enhancing Document Layout Analysis through Diverse Synthetic Data and Global-to-Local Adaptive Perceptioncited by 82
  12. 2023Vision Grid Transformer for Document Layout Analysiscited by 47
38 more
  1. 2024Document Parsing Unveiled: Techniques, Challenges, and Prospects for Structured Information Extractioncited by 44
  2. 2023StrucTexTv2: Masked Visual-Textual Prediction for Document Image Pre-trainingcited by 39
  3. 2023ICL-D3IE: In-Context Learning with Diverse Demonstrations Updating for Document Information Extractioncited by 38
  4. 2024Universal Physics Transformers: A Framework For Efficiently Scaling Neural Operatorscited by 27
  5. 2025PP-DocLayout: A Unified Document Layout Detection Model to Accelerate Large-Scale Data Constructioncited by 26
  6. 2024LayoutLLM: Large Language Model Instruction Tuning for Visually Rich Document Understandingcited by 24
  7. 2023Reading Order Matters: Information Extraction from Visually-rich Documents by Token Path Predictioncited by 23
  8. 2024Deep Learning based Visually Rich Document Content Understanding: A Surveycited by 21
  9. 2022Wukong-Reader: Multi-modal Pre-training for Fine-grained Visual Document Understandingcited by 20
  10. 2023FormNetV2: Multimodal Graph Contrastive Learning for Form Document Information Extractioncited by 20
  11. 2023LayoutMask: Enhance Text-Layout Interaction in Multi-modal Pre-training for Document Understandingcited by 17
  12. 2023Privacy-Aware Document Visual Question Answeringcited by 17
  13. 2022Evaluating Out-of-Distribution Performance on Document Image Classifierscited by 16
  14. 2022Deep learning for table detection and structure recognition: A surveycited by 15
  15. 2024Detect-Order-Construct: A Tree Construction based Approach for Hierarchical Document Structure Analysiscited by 15
  16. 2024Diff-Restorer: Unleashing Visual Prompts for Diffusion-based Universal Image Restorationcited by 15
  17. 2022VLCDoC: Vision-Language Contrastive Pre-Training Model for Cross-Modal Document Classificationcited by 14
  18. 2024SciPostLayout: A Dataset for Layout Analysis and Layout Generation of Scientific Posterscited by 14
  19. 2023SwinDocSegmenter: An End-to-End Unified Domain Adaptive Transformer for Document Instance Segmentationcited by 13
  20. 2024RoDLA: Benchmarking the Robustness of Document Layout Analysis Modelscited by 13
  21. 2022XDoc: Unified Pre-training for Cross-Format Document Understandingcited by 12
  22. 2024DLAFormer: An End-to-End Transformer For Document Layout Analysiscited by 11
  23. 2023Beyond Document Page Classification: Design, Datasets, and Challengescited by 10
  24. 2023A Graphical Approach to Document Layout Analysiscited by 8
  25. 2024Multi-Page Document Visual Question Answering using Self-Attention Scoring Mechanismcited by 8
  26. 2024TRINS: Towards Multimodal Language Models that Can Readcited by 8
  27. 2022Page Layout Analysis of Text-heavy Historical Documents: a Comparison of Textual and Visual Approachescited by 7
  28. 2023SelfDocSeg: A Self-Supervised vision-based Approach towards Document Segmentationcited by 7
  29. 2023Bridging the Performance Gap between DETR and R-CNN for Graphical Object Detection in Document Imagescited by 7
  30. 2023DSG: An End-to-End Document Structure Generatorcited by 7
  31. 2023WordScape: a Pipeline to extract multilingual, visually rich Documents with Layout Annotations from Web Crawl Datacited by 7
  32. 2024μgat: Improving Single-Page Document Parsing by Providing Multi-Page Contextcited by 7
  33. 2023ICDAR 2023 Competition on Robust Layout Segmentation in Corporate Documentscited by 6
  34. 2023Do-GOOD: Towards Distribution Shift Evaluation for Pre-Trained Visual Document Understanding Modelscited by 6
  35. 2024Transformers and Language Models in Form Understanding: A Comprehensive Review of Scanned Document Analysiscited by 6
  36. 2024A Hybrid Approach for Document Layout Analysis in Document imagescited by 6
  37. 2024Deep Learning based Key Information Extraction from Business Documents: Systematic Literature Reviewcited by 6
  38. 2024MMR: Evaluating Reading Ability of Large Multimodal Modelscited by 6

50 most cited of 152 All 152 (JSON)