Label Errors in the Tobacco3482 Dataset
Not on the map. Why?
Abstract (arXiv)
Tobacco3482 is a widely used document classification benchmark dataset. However, our manual inspection of the entire dataset uncovers widespread ontological issues, especially large amounts of annotation label problems in the dataset. We establish data label guidelines and find that 11.7% of the dataset is improperly annotated and should either have an unknown label or a corrected label, and 16.7% of samples in the dataset have multiple valid labels. We then analyze the mistakes of a top-performing model and find that 35% of the model's mistakes can be directly attributed to these label issues, highlighting the inherent problems with using a noisily labeled dataset as a benchmark. Supplementary material, including dataset annotations and code, is available at https://github.com/gordon-lim/tobacco3482-mistakes/.
Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 7 of its 21 references that are in the map. First posted on arXiv 17 Dec 2024.
Builds on
7 of 21 references.
References · 7
- 2019Confident Learning: Estimating Uncertainty in Dataset Labelscited by 411
- 2021Pervasive Label Errors in Test Sets Destabilize Machine Learning Benchmarkscited by 388
- 2015Evaluation of Deep Convolutional Nets for Document Image Classification and Retrievalcited by 212
- 2022DiT: Self-supervised Pre-training for Document Image Transformercited by 152
- 2019Do We Train on Test Data? Purging CIFAR of Near-Duplicatescited by 61
- 2023Do We Train on Test Data? The Impact of Near-Duplicates on License Plate Recognitioncited by 14
- 2023On Evaluation of Document Classification using RVL-CDIPcited by 5
Built on by
Not cited yet.