# Revising RVL-CDIP: Quantifying Errors and Test-Train Overlap

arXiv: https://arxiv.org/abs/2606.31446 · HTML: https://mlsanity.com/paper/2606.31446/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 34 of its 61 references that are in the map.
- First posted on arXiv 30 Jun 2026.

## Builds on (most cited first, 30 of 34)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Very Deep Convolutional Networks for Large-Scale Image Recognition](https://mlsanity.com/paper/1409.1556/) (2014)
- [Going Deeper with Convolutions](https://mlsanity.com/paper/1409.4842/) (2014)
- [Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervision](https://mlsanity.com/paper/2102.05918/) (2021)
- [Longformer: The Long-Document Transformer](https://mlsanity.com/paper/2004.05150/) (2020)
- [Aggregated Residual Transformations for Deep Neural Networks](https://mlsanity.com/paper/1611.05431/) (2016)
- [Big Bird: Transformers for Longer Sequences](https://mlsanity.com/paper/2007.14062/) (2020)
- [gpt-oss-120b & gpt-oss-20b Model Card](https://mlsanity.com/paper/2508.10925/) (2025)
- [SuperPoint: Self-Supervised Interest Point Detection and Description](https://mlsanity.com/paper/1712.07629/) (2017)
- [SigLIP 2: Multilingual Vision-Language Encoders with Improved Semantic Understanding, Localization, and Dense Features](https://mlsanity.com/paper/2502.14786/) (2025)
- [LayoutLM: Pre-training of Text and Layout for Document Image Understanding](https://mlsanity.com/paper/1912.13318/) (2019)
- [LayoutLMv3: Pre-training for Document AI with Unified Text and Image Masking](https://mlsanity.com/paper/2204.08387/) (2022)
- [Pervasive Label Errors in Test Sets Destabilize Machine Learning Benchmarks](https://mlsanity.com/paper/2103.14749/) (2021)
- [Evaluation of Deep Convolutional Nets for Document Image Classification and Retrieval](https://mlsanity.com/paper/1502.07058/) (2015)
- [Revisiting Oxford and Paris: Large-Scale Image Retrieval Benchmarking](https://mlsanity.com/paper/1803.11285/) (2018)
- [DiT: Self-supervised Pre-training for Document Image Transformer](https://mlsanity.com/paper/2203.02378/) (2022)
- [Unifying Vision, Text, and Layout for Universal Document Processing](https://mlsanity.com/paper/2212.02623/) (2022)
- [Question and Answer Test-Train Overlap in Open-Domain Question Answering Datasets](https://mlsanity.com/paper/2008.02637/) (2020)
- [The Adverse Effects of Code Duplication in Machine Learning Models of Code](https://mlsanity.com/paper/1812.06469/) (2018)
- [Do We Train on Test Data? Purging CIFAR of Near-Duplicates](https://mlsanity.com/paper/1902.00423/) (2019)
- [Data Quality for Software Vulnerability Datasets](https://mlsanity.com/paper/2301.05456/) (2023)
- [Identifying Mislabeled Instances in Classification Datasets](https://mlsanity.com/paper/1912.05283/) (2019)
- [LayoutLLM: Large Language Model Instruction Tuning for Visually Rich Document Understanding](https://mlsanity.com/paper/2403.14252/) (2024)
- [Bi-VLDoc: Bidirectional Vision-Language Modeling for Visually-Rich Document Understanding](https://mlsanity.com/paper/2206.13155/) (2022)
- [Evaluating Out-of-Distribution Performance on Document Image Classifiers](https://mlsanity.com/paper/2210.07448/) (2022)
- [Detecting Label Errors in Token Classification Data](https://mlsanity.com/paper/2210.03920/) (2022)
- [CleanCoNLL: A Nearly Noise-Free Named Entity Recognition Dataset](https://mlsanity.com/paper/2310.16225/) (2023)
- [Revising FUNSD dataset for key-value detection in document images](https://mlsanity.com/paper/2010.05322/) (2020)
- [Information Redundancy and Biases in Public Document Information Extraction Benchmarks](https://mlsanity.com/paper/2304.14936/) (2023)

## Abstract (arXiv)

RVL-CDIP is a popular dataset for benchmarking document classifiers. However, the dataset contains ample amounts of label errors as well as non-trivial amounts of test-train overlap, both of which may impact model performance metrics. In this paper, we address these two problems by (1) finding and fixing label errors, and (2) detecting and addressing test-train overlap. We produce several variations of RVL-CDIP with label error and test-train overlap fixes, and benchmark document classification performance on these new RVL-CDIP variations. Our rigorous analysis of RVL-CDIP finds that the corpus contains 12% label error and approximately 35% test-train duplication. Remediation sees improvements in classification accuracy when errors are removed, but sees decreases in accuracy when duplicates are removed. We additionally evaluate models on RVL-CDIP-N, an out-of-distribution benchmark, finding that training on error-corrected data substantially improves OOD generalization, with supervised models gaining an average of 8.1 percentage points in accuracy and improvements as large as 14 percentage points.