mlsanityData to 14 Oct 2026

On Evaluation of Document Classification using RVL-CDIP

arXiv 2306.12550 ↗First posted 21 Jun 2023cs.CLcited by 5

Not on the map. Why?

Abstract (arXiv)

The RVL-CDIP benchmark is widely used for measuring performance on the task of document classification. Despite its widespread use, we reveal several undesirable characteristics of the RVL-CDIP benchmark. These include (1) substantial amounts of label noise, which we estimate to be 8.1% (ranging between 1.6% to 16.9% per document category); (2) presence of many ambiguous or multi-label documents; (3) a large overlap between test and train splits, which can inflate model performance metrics; and (4) presence of sensitive personally-identifiable information like US Social Security numbers (SSNs). We argue that there is a risk in using RVL-CDIP for benchmarking document classifiers, as its limited scope, presence of errors (state-of-the-art models now achieve accuracy error rates that are within our estimated label error rate), and lack of diversity make it less than ideal for benchmarking. We further advocate for the creation of a new document classification benchmark, and provide recommendations for what characteristics such a resource should include.

Key facts. Cited by 5 papers in the map (data through 14 Oct 2026). Builds on 65 of its 89 references that are in the map. First posted on arXiv 21 Jun 2023.

Builds on

65 of 89 references.

References · 65

  1. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  3. 2014Very Deep Convolutional Networks for Large-Scale Image Recognitioncited by 15,364
  4. 2014Going Deeper with Convolutionscited by 4,241
  5. 2020Extracting Training Data from Large Language Modelscited by 2,189
  6. 2019Do ImageNet Classifiers Generalize to ImageNet?cited by 1,515
  7. 2020WILDS: A Benchmark of in-the-Wild Distribution Shiftscited by 1,375
  8. 2020DocVQA: A Dataset for VQA on Document Imagescited by 1,317
  9. 2019Right for the Wrong Reasons: Diagnosing Syntactic Heuristics in Natural Language Inferencecited by 876
  10. 2018Annotation Artifacts in Natural Language Inference Datacited by 671
  11. 2019LayoutLM: Pre-training of Text and Layout for Document Image Understandingcited by 553
  12. 2022LayoutLMv3: Pre-training for Document AI with Unified Text and Image Maskingcited by 485
38 more
  1. 2019Confident Learning: Estimating Uncertainty in Dataset Labelscited by 413
  2. 2020LayoutLMv2: Multi-modal Pre-training for Visually-Rich Document Understandingcited by 408
  3. 2021OCR-free Document Understanding Transformercited by 407
  4. 2021Pervasive Label Errors in Test Sets Destabilize Machine Learning Benchmarkscited by 395
  5. 2019FUNSD: A Dataset for Form Understanding in Noisy Scanned Documentscited by 349
  6. 2020Identifying Mislabeled Data using the Area Under the Margin Rankingcited by 261
  7. 2021DocFormer: End-to-End Transformer for Document Understandingcited by 256
  8. 2021Quality at a Glance: An Audit of Web-Crawled Multilingual Datasetscited by 243
  9. 2015Evaluation of Deep Convolutional Nets for Document Image Classification and Retrievalcited by 213
  10. 2018Revisiting Oxford and Paris: Large-Scale Image Retrieval Benchmarkingcited by 191
  11. 2022DiT: Self-supervised Pre-training for Document Image Transformercited by 152
  12. 2022Unifying Vision, Text, and Layout for Universal Document Processingcited by 140
  13. 2020Question and Answer Test-Train Overlap in Open-Domain Question Answering Datasetscited by 138
  14. 2018Improving Text-to-SQL Evaluation Methodologycited by 135
  15. 2021Going Full-TILT Boogie on Document Understanding with Text-Image-Layout Transformercited by 134
  16. 2018The Adverse Effects of Code Duplication in Machine Learning Models of Codecited by 126
  17. 2021SelfDoc: Self-Supervised Document Representation Learningcited by 117
  18. 2022LiLT: A Simple yet Effective Language-Independent Layout Transformer for Structured Document Understandingcited by 107
  19. 2021Shifts: A Dataset of Real Distributional Shift Across Multiple Large-Scale Taskscited by 101
  20. 2021StructuralLM: Structural Pre-training for Form Understandingcited by 91
  21. 2022Unified Pretraining Framework for Document Understandingcited by 85
  22. 2022End-to-end Document Recognition and Understanding with Dessurtcited by 73
  23. 2016A Thorough Examination of the CNN/Daily Mail Reading Comprehension Taskcited by 72
  24. 2020We Need to Talk About Random Splitscited by 71
  25. 2021Memorization vs. Generalization: Quantifying Data Leakage in NLP Performance Evaluationcited by 70
  26. 2022ERNIE-Layout: Layout Knowledge Enhanced Pre-training for Visually-rich Document Understandingcited by 67
  27. 2019Do We Train on Test Data? Purging CIFAR of Near-Duplicatescited by 62
  28. 2023Data Quality for Software Vulnerability Datasetscited by 51
  29. 2023StrucTexTv2: Masked Visual-Textual Prediction for Document Image Pre-trainingcited by 39
  30. 2022OCR-IDL: OCR Annotations for Industry Document Library Datasetcited by 35
  31. 2019Outlier Detection for Improved Data Quality and Diversity in Dialog Systemscited by 28
  32. 2019Identifying Mislabeled Instances in Classification Datasetscited by 26
  33. 2017Generalisation in Named Entity Recognition: A Quantitative Analysiscited by 25
  34. 2019Multimodal deep networks for text and image-based document classificationcited by 24
  35. 2017Cutting the Error by Half: Investigation of Very Deep CNN and Advanced Training Strategies for Document Image Classificationcited by 22
  36. 2020Towards a Multi-modal, Multi-task Learning based Pre-training Framework for Document Representation Learningcited by 21
  37. 2018Document Image Classification with Intra-Domain Transfer Learning and Stacked Generalization of Deep Convolutional Neural Networkscited by 20
  38. 2022Wukong-Reader: Multi-modal Pre-training for Fine-grained Visual Document Understandingcited by 20

50 most cited of 65 All 65 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 2; 2025: 0; 2026: 3.

5 papers.

  1. 2026DocSplit: A Comprehensive Benchmark Dataset and Evaluation Approach for Document Packet Recognition and Splittingcited by 5
  2. 2024DistilDoc: Knowledge Distillation for Visually-Rich Document Applicationscited by 1
  3. 2024Label Errors in the Tobacco3482 Datasetcited by 0
  4. 2026Multimodal Approaches for Visually-Rich Document Type Classification: A Comparative Analysiscited by 0
  5. 2026When Superpixels Fail on Documents: A Study of Segmentation for LIME Explanationscited by 0