mlsanityData to 7 Oct 2026

Evaluating Out-of-Distribution Performance on Document Image Classifiers

arXiv 2210.07448 ↗First posted 14 Oct 2022cs.CV cs.CLcited by 16

Not on the map. Why?

Abstract (arXiv)

The ability of a document classifier to handle inputs that are drawn from a distribution different from the training distribution is crucial for robust deployment and generalizability. The RVL-CDIP corpus is the de facto standard benchmark for document classification, yet to our knowledge all studies that use this corpus do not include evaluation on out-of-distribution documents. In this paper, we curate and release a new out-of-distribution benchmark for evaluating out-of-distribution performance for document classifiers. Our new out-of-distribution benchmark consists of two types of documents: those that are not part of any of the 16 in-domain RVL-CDIP categories (RVL-CDIP-O), and those that are one of the 16 in-domain categories yet are drawn from a distribution different from that of the original RVL-CDIP dataset (RVL-CDIP-N). While prior work on document classification for in-domain RVL-CDIP documents reports high accuracy scores, we find that these models exhibit accuracy drops of between roughly 15-30% on our new out-of-domain RVL-CDIP-N benchmark, and further struggle to distinguish between in-domain RVL-CDIP-N and out-of-domain RVL-CDIP-O inputs. Our new benchmark provides researchers with a valuable new resource for analyzing out-of-distribution performance on document classifiers. Our new out-of-distribution data can be found at https://github.com/gxlarson/rvl-cdip-ood.

Key facts. Cited by 16 papers in the map (data through 7 Oct 2026). Builds on 50 of its 61 references that are in the map. First posted on arXiv 14 Oct 2022.

Builds on

50 of 61 references.

References · 50

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
  3. 2014Very Deep Convolutional Networks for Large-Scale Image Recognitioncited by 15,267
  4. 2014Going Deeper with Convolutionscited by 4,231
  5. 2019Benchmarking Neural Network Robustness to Common Corruptions and Perturbationscited by 2,857
  6. 2016A Baseline for Detecting Misclassified and Out-of-Distribution Examples in Neural Networkscited by 2,362
  7. 2021BEiT: BERT Pre-Training of Image Transformerscited by 2,360
  8. 2019Natural Adversarial Examplescited by 1,401
  9. 2020WILDS: A Benchmark of in-the-Wild Distribution Shiftscited by 1,366
  10. 2018Moment Matching for Multi-Source Domain Adaptationcited by 1,327
  11. 2017Enhancing The Reliability of Out-of-distribution Image Detection in Neural Networkscited by 1,325
  12. 2020Energy-based Out-of-distribution Detectioncited by 1,296
38 more
  1. 2017Deep Hashing Network for Unsupervised Domain Adaptationcited by 1,181
  2. 2020Beyond Accuracy: Behavioral Testing of NLP models with CheckListcited by 979
  3. 2019LayoutLM: Pre-training of Text and Layout for Document Image Understandingcited by 549
  4. 2020Generalized ODIN: Detecting Out-of-distribution Image without Learning from Out-of-distribution Datacited by 408
  5. 2020LayoutLMv2: Multi-modal Pre-training for Visually-Rich Document Understandingcited by 406
  6. 2019An Evaluation Dataset for Intent Classification and Out-of-Scope Predictioncited by 378
  7. 2020Pretrained Transformers Improve Out-of-Distribution Robustnesscited by 293
  8. 2021DocFormer: End-to-End Transformer for Document Understandingcited by 254
  9. 2021Understanding Robustness of Transformers for Image Classificationcited by 250
  10. 2015Evaluation of Deep Convolutional Nets for Document Image Classification and Retrievalcited by 212
  11. 2021Vision Transformers are Robust Learnerscited by 208
  12. 2021Are Transformers More Robust Than CNNs?cited by 187
  13. 2021On the Robustness of Vision Transformers to Adversarial Examplescited by 167
  14. 2022DiT: Self-supervised Pre-training for Document Image Transformercited by 152
  15. 2021Going Full-TILT Boogie on Document Understanding with Text-Image-Layout Transformercited by 134
  16. 2022LiLT: A Simple yet Effective Language-Independent Layout Transformer for Structured Document Understandingcited by 107
  17. 2021StructuralLM: Structural Pre-training for Form Understandingcited by 91
  18. 2022Unified Pretraining Framework for Document Understandingcited by 85
  19. 2022End-to-end Document Recognition and Understanding with Dessurtcited by 73
  20. 2022ERNIE-Layout: Layout Knowledge Enhanced Pre-training for Visually-rich Document Understandingcited by 67
  21. 2022Can CNNs Be More Robust Than Transformers?cited by 38
  22. 2020RADDLE: An Evaluation Benchmark and Analysis Platform for Robust Task-oriented Dialog Systemscited by 36
  23. 2022AugLy: Data Augmentations for Robustnesscited by 34
  24. 2019Multimodal deep networks for text and image-based document classificationcited by 24
  25. 2017Cutting the Error by Half: Investigation of Very Deep CNN and Advanced Training Strategies for Document Image Classificationcited by 22
  26. 2020Towards a Multi-modal, Multi-task Learning based Pre-training Framework for Document Representation Learningcited by 21
  27. 2018Document Image Classification with Intra-Domain Transfer Learning and Stacked Generalization of Deep Convolutional Neural Networkscited by 20
  28. 2019Modular Multimodal Architecture for Document Classificationcited by 15
  29. 2017Real-Time Document Image Classification using Deep CNN and Extreme Learning Machinescited by 14
  30. 2022VLCDoC: Vision-Language Contrastive Pre-Training Model for Cross-Modal Document Classificationcited by 14
  31. 2022Augraphy: A Data Augmentation Library for Document Imagescited by 14
  32. 2017Analysis of Convolutional Neural Networks for Document Image Classificationcited by 13
  33. 2020Improving accuracy and speeding up Document Image Classification through parallel systemscited by 11
  34. 2021Natural Adversarial Objectscited by 7
  35. 2022Understanding Long Documents with Different Position-Aware Attentionscited by 7
  36. 2022Redwood: Using Collision Detection to Grow a Large-Scale Intent Classification Datasetcited by 6
  37. 2021Class-wise Thresholding for Robust Out-of-Distribution Detectioncited by 3
  38. 2022MATrIX -- Modality-Aware Transformer for Information eXtractioncited by 2

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 1; 2023: 7; 2024: 4; 2025: 0; 2026: 4.

16 papers.

Sort:

  1. 2022GLUE-X: Evaluating Natural Language Understanding Models from an Out-of-distribution Generalization Perspectivecited by 72
  2. 2023Supervised Knowledge Makes Large Language Models Better In-context Learnerscited by 20
  3. 2023Beyond Document Page Classification: Design, Datasets, and Challengescited by 10
  4. 2023Do-GOOD: Towards Distribution Shift Evaluation for Pre-Trained Visual Document Understanding Modelscited by 6
  5. 2023On Evaluation of Document Classification using RVL-CDIPcited by 5
  6. 2026DocSplit: A Comprehensive Benchmark Dataset and Evaluation Approach for Document Packet Recognition and Splittingcited by 5
  7. 2023GeSS: Benchmarking Geometric Deep Learning under Scientific Applications with Distribution Shiftscited by 4
  8. 2024DocKD: Knowledge Distillation from LLMs for Open-World Document Understanding Modelscited by 3
  9. 2023Diving into the Depths of Spotting Text in Multi-Domain Noisy Scenescited by 2
  10. 2024Generating Hard-Negative Out-of-Scope Data with ChatGPT for Intent Classificationcited by 2
  11. 2026COMBOOD: A Semiparametric Approach for Detecting Out-of-distribution Data for Image Classificationcited by 2
  12. 2023GVdoc: Graph-based Visual Document Classificationcited by 1
4 more
  1. 2024DistilDoc: Knowledge Distillation for Visually-Rich Document Applicationscited by 1
  2. 2024Document Classification using File Namescited by 1
  3. 2026Multi-domain Multi-modal Document Classification Benchmark with a Multi-level Taxonomycited by 0
  4. 2026Revising RVL-CDIP: Quantifying Errors and Test-Train Overlapcited by 0