mlsanityData to 7 Oct 2026

Revising RVL-CDIP: Quantifying Errors and Test-Train Overlap

arXiv 2606.31446 ↗First posted 30 Jun 2026cs.CL cs.CVcited by 0

Not on the map. Why?

Abstract (arXiv)

RVL-CDIP is a popular dataset for benchmarking document classifiers. However, the dataset contains ample amounts of label errors as well as non-trivial amounts of test-train overlap, both of which may impact model performance metrics. In this paper, we address these two problems by (1) finding and fixing label errors, and (2) detecting and addressing test-train overlap. We produce several variations of RVL-CDIP with label error and test-train overlap fixes, and benchmark document classification performance on these new RVL-CDIP variations. Our rigorous analysis of RVL-CDIP finds that the corpus contains 12% label error and approximately 35% test-train duplication. Remediation sees improvements in classification accuracy when errors are removed, but sees decreases in accuracy when duplicates are removed. We additionally evaluate models on RVL-CDIP-N, an out-of-distribution benchmark, finding that training on error-corrected data substantially improves OOD generalization, with supervised models gaining an average of 8.1 percentage points in accuracy and improvements as large as 14 percentage points.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 34 of its 61 references that are in the map. First posted on arXiv 30 Jun 2026.

Builds on

34 of 61 references.

References · 34

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  3. 2014Very Deep Convolutional Networks for Large-Scale Image Recognitioncited by 15,267
  4. 2014Going Deeper with Convolutionscited by 4,231
  5. 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,848
  6. 2020Longformer: The Long-Document Transformercited by 3,362
  7. 2016Aggregated Residual Transformations for Deep Neural Networkscited by 2,540
  8. 2020Big Bird: Transformers for Longer Sequencescited by 1,673
  9. 2017SuperPoint: Self-Supervised Interest Point Detection and Descriptioncited by 1,313
  10. 2025gpt-oss-120b & gpt-oss-20b Model Cardcited by 1,258
  11. 2025SigLIP 2: Multilingual Vision-Language Encoders with Improved Semantic Understanding, Localization, and Dense Featurescited by 1,127
  12. 2019LayoutLM: Pre-training of Text and Layout for Document Image Understandingcited by 549
22 more
  1. 2022LayoutLMv3: Pre-training for Document AI with Unified Text and Image Maskingcited by 481
  2. 2021Pervasive Label Errors in Test Sets Destabilize Machine Learning Benchmarkscited by 388
  3. 2015Evaluation of Deep Convolutional Nets for Document Image Classification and Retrievalcited by 212
  4. 2018Revisiting Oxford and Paris: Large-Scale Image Retrieval Benchmarkingcited by 189
  5. 2022DiT: Self-supervised Pre-training for Document Image Transformercited by 152
  6. 2022Unifying Vision, Text, and Layout for Universal Document Processingcited by 140
  7. 2020Question and Answer Test-Train Overlap in Open-Domain Question Answering Datasetscited by 137
  8. 2018The Adverse Effects of Code Duplication in Machine Learning Models of Codecited by 123
  9. 2019Do We Train on Test Data? Purging CIFAR of Near-Duplicatescited by 61
  10. 2023Data Quality for Software Vulnerability Datasetscited by 49
  11. 2019Identifying Mislabeled Instances in Classification Datasetscited by 26
  12. 2024LayoutLLM: Large Language Model Instruction Tuning for Visually Rich Document Understandingcited by 24
  13. 2022Bi-VLDoc: Bidirectional Vision-Language Modeling for Visually-Rich Document Understandingcited by 18
  14. 2022Evaluating Out-of-Distribution Performance on Document Image Classifierscited by 16
  15. 2022Detecting Label Errors in Token Classification Datacited by 12
  16. 2023CleanCoNLL: A Nearly Noise-Free Named Entity Recognition Datasetcited by 11
  17. 2020Revising FUNSD dataset for key-value detection in document imagescited by 10
  18. 2023Information Redundancy and Biases in Public Document Information Extraction Benchmarkscited by 4
  19. 2023EAML: Ensemble Self-Attention-based Mutual Learning Network for Document Image Classificationcited by 4
  20. 2023Impact of Ground Truth Quality on Handwriting Recognitioncited by 3
  21. 2024Identity Overlap Between Face Recognition Train/Test Data: Causing Optimistic Bias in Accuracy Measurementcited by 3
  22. 2024Enhancing Data Quality through Simple De-duplication: Navigating Responsible Computational Social Science Researchcited by 1

Built on by

Not cited yet.