# Assessing the Quality of the Datasets by Identifying Mislabeled Samples

arXiv: https://arxiv.org/abs/2109.05000 · HTML: https://mlsanity.com/paper/2109.05000/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Builds on 27 of its 38 references that are in the map.
- First posted on arXiv 10 Sep 2021.

## Builds on (most cited first, 27 of 27)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Very Deep Convolutional Networks for Large-Scale Image Recognition](https://mlsanity.com/paper/1409.1556/) (2014)
- [Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithms](https://mlsanity.com/paper/1708.07747/) (2017)
- [Auto-Encoding Variational Bayes](https://mlsanity.com/paper/1312.6114/) (2013)
- [An Introduction to Variational Autoencoders](https://mlsanity.com/paper/1906.02691/) (2019)
- [A Closer Look at Memorization in Deep Networks](https://mlsanity.com/paper/1706.05394/) (2017)
- [Learning to Reweight Examples for Robust Deep Learning](https://mlsanity.com/paper/1803.09050/) (2018)
- [Making Deep Neural Networks Robust to Label Noise: a Loss Correction Approach](https://mlsanity.com/paper/1609.03683/) (2016)
- [Learning from Noisy Labels with Deep Neural Networks: A Survey](https://mlsanity.com/paper/2007.08199/) (2020)
- [Fine-Grained Analysis of Optimization and Generalization for Overparameterized Two-Layer Neural Networks](https://mlsanity.com/paper/1901.08584/) (2019)
- [Semi-Supervised Learning with Deep Generative Models](https://mlsanity.com/paper/1406.5298/) (2014)
- [Dataset Cartography: Mapping and Diagnosing Datasets with Training Dynamics](https://mlsanity.com/paper/2009.10795/) (2020)
- [Training Deep Neural Networks on Noisy Labels with Bootstrapping](https://mlsanity.com/paper/1412.6596/) (2014)
- [Joint Optimization Framework for Learning with Noisy Labels](https://mlsanity.com/paper/1803.11364/) (2018)
- [WebVision Database: Visual Learning and Understanding from Web Data](https://mlsanity.com/paper/1708.02862/) (2017)
- [Identifying Mislabeled Data using the Area Under the Margin Ranking](https://mlsanity.com/paper/2001.10528/) (2020)
- [Decoupling "when to update" from "how to update"](https://mlsanity.com/paper/1706.02613/) (2017)
- [Understanding and Utilizing Deep Neural Networks Trained with Noisy Labels](https://mlsanity.com/paper/1905.05040/) (2019)
- [Dimensionality-Driven Learning with Noisy Labels](https://mlsanity.com/paper/1806.02612/) (2018)
- [Learning from Noisy Labels with Distillation](https://mlsanity.com/paper/1703.02391/) (2017)
- [Learning to Learn from Noisy Labeled Data](https://mlsanity.com/paper/1812.05214/) (2018)
- [Image Classification with Deep Learning in the Presence of Noisy Labels: A Survey](https://mlsanity.com/paper/1912.05170/) (2019)
- [Anomaly Detection With Conditional Variational Autoencoders](https://mlsanity.com/paper/2010.05531/) (2020)
- [Robust Anomaly Detection in Images using Adversarial Autoencoders](https://mlsanity.com/paper/1901.06355/) (2019)
- [Identifying Mislabeled Instances in Classification Datasets](https://mlsanity.com/paper/1912.05283/) (2019)
- [Embracing Error to Enable Rapid Crowdsourcing](https://mlsanity.com/paper/1602.04506/) (2016)
- [Augmenting Variational Autoencoders with Sparse Labels: A Unified Framework for Unsupervised, Semi-(un)supervised, and Supervised Learning](https://mlsanity.com/paper/1908.03015/) (2019)

## Built on by (most cited first, 5 of 5)

- [Towards Explainable Artificial Intelligence (XAI): A Data Mining Perspective](https://mlsanity.com/paper/2401.04374/) (2024)
- [Noise Audits Improve Moral Foundation Classification](https://mlsanity.com/paper/2210.07415/) (2022)
- [Improving Data Quality with Training Dynamics of Gradient Boosting Decision Trees](https://mlsanity.com/paper/2210.11327/) (2022)
- [Leveraging Unlabeled Data to Track Memorization](https://mlsanity.com/paper/2212.04461/) (2022)
- [Do-AIQ: A Design-of-Experiment Approach to Quality Evaluation of AI Mislabel Detection Algorithm](https://mlsanity.com/paper/2208.09953/) (2022)

## Abstract (arXiv)

Due to the over-emphasize of the quantity of data, the data quality has often been overlooked. However, not all training data points contribute equally to learning. In particular, if mislabeled, it might actively damage the performance of the model and the ability to generalize out of distribution, as the model might end up learning spurious artifacts present in the dataset. This problem gets compounded by the prevalence of heavily parameterized and complex deep neural networks, which can, with their high capacity, end up memorizing the noise present in the dataset. This paper proposes a novel statistic -- noise score, as a measure for the quality of each data point to identify such mislabeled samples based on the variations in the latent space representation. In our work, we use the representations derived by the inference network of data quality supervised variational autoencoder (AQUAVS). Our method leverages the fact that samples belonging to the same class will have similar latent representations. Therefore, by identifying the outliers in the latent space, we can find the mislabeled samples. We validate our proposed statistic through experimentation by corrupting MNIST, FashionMNIST, and CIFAR10/100 datasets in different noise settings for the task of identifying mislabelled samples. We further show significant improvements in accuracy for the classification task for each dataset.