# Standardness Clouds Meaning: A Position Regarding the Informed Usage of Standard Datasets

arXiv: https://arxiv.org/abs/2406.13552 · HTML: https://mlsanity.com/paper/2406.13552/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 22 of its 85 references that are in the map.
- First posted on arXiv 19 Jun 2024.

## Builds on (most cited first, 22 of 22)

- [A Survey on Bias and Fairness in Machine Learning](https://mlsanity.com/paper/1908.09635/) (2019)
- [Datasheets for Datasets](https://mlsanity.com/paper/1803.09010/) (2018)
- [Confident Learning: Estimating Uncertainty in Dataset Labels](https://mlsanity.com/paper/1911.00068/) (2019)
- [Understanding and Utilizing Deep Neural Networks Trained with Noisy Labels](https://mlsanity.com/paper/1905.05040/) (2019)
- [Towards Explainable Artificial Intelligence](https://mlsanity.com/paper/1909.12072/) (2019)
- [Hierarchical Transformers for Long Document Classification](https://mlsanity.com/paper/1910.10781/) (2019)
- [The Dataset Nutrition Label: A Framework To Drive Higher Data Quality Standards](https://mlsanity.com/paper/1805.03677/) (2018)
- [Mitigating backdoor attacks in LSTM-based Text Classification Systems by Backdoor Keyword Identification](https://mlsanity.com/paper/2007.12070/) (2020)
- [Cold Case: The Lost MNIST Digits](https://mlsanity.com/paper/1905.10498/) (2019)
- [Morpho-MNIST: Quantitative Assessment and Diagnostics for Representation Learning](https://mlsanity.com/paper/1809.10780/) (2018)
- [A Topological "Reading" Lesson: Classification of MNIST using TDA](https://mlsanity.com/paper/1910.08345/) (2019)
- [Measuring Data](https://mlsanity.com/paper/2212.05129/) (2022)
- [A domain-specific language for describing machine learning datasets](https://mlsanity.com/paper/2207.02848/) (2022)
- [Classes are not Clusters: Improving Label-based Evaluation of Dimensionality Reduction](https://mlsanity.com/paper/2308.00278/) (2023)
- [Assessing four Neural Networks on Handwritten Digit Recognition Dataset (MNIST)](https://mlsanity.com/paper/1811.08278/) (2018)
- [Large-Scale Evaluation of Topic Models and Dimensionality Reduction Methods for 2D Text Spatialization](https://mlsanity.com/paper/2307.11770/) (2023)
- [Ensuring Dataset Quality for Machine Learning Certification](https://mlsanity.com/paper/2011.01799/) (2020)
- [HardVis: Visual Analytics to Handle Instance Hardness Using Undersampling and Oversampling Techniques](https://mlsanity.com/paper/2203.15753/) (2022)
- [Semi-supervised deep learning based on label propagation in a 2D embedded space](https://mlsanity.com/paper/2008.00558/) (2020)
- [Incorporation of Human Knowledge into Data Embeddings to Improve Pattern Significance and Interpretability](https://mlsanity.com/paper/2209.11364/) (2022)
- [Dataset Definition Standard (DDS)](https://mlsanity.com/paper/2101.03020/) (2021)
- [A Robust Hybrid Approach for Textual Document Classification](https://mlsanity.com/paper/1909.05478/) (2019)

## Abstract (arXiv)

Standard datasets are frequently used to train and evaluate Machine Learning models. However, the assumed standardness of these datasets leads to a lack of in-depth discussion on how their labels match the derived categories for the respective use case, which we demonstrate by reviewing recent literature that employs standard datasets. We find that the standardness of the datasets seems to cloud their actual coherency and applicability, thus impeding the trust in Machine Learning models trained on these datasets. Therefore, we argue against the uncritical use of standard datasets and advocate for their critical examination instead. For this, we suggest to use Grounded Theory in combination with Hypotheses Testing through Visualization as methods to evaluate the match between use case, derived categories, and labels. We exemplify this approach by applying it to the 20 Newsgroups dataset and the MNIST dataset, both considered standard datasets in their respective domain. The results show that the labels of the 20 Newsgroups dataset are imprecise, which implies that neither a Machine Learning model can learn a meaningful abstraction of derived categories nor one can draw conclusions from achieving high accuracy on this dataset. For the MNIST dataset, we demonstrate that the labels can be confirmed to be defined well. We conclude that also for datasets that are considered to be standard, quality and suitability have to be assessed in order to learn meaningful abstractions and, thus, improve trust in Machine Learning models.