# A Robust Optimization Method for Label Noisy Datasets Based on Adaptive Threshold: Adaptive-k

arXiv: https://arxiv.org/abs/2203.14165 · HTML: https://mlsanity.com/paper/2203.14165/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 11 of its 26 references that are in the map.
- First posted on arXiv 26 Mar 2022.

## Builds on (most cited first, 11 of 11)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithms](https://mlsanity.com/paper/1708.07747/) (2017)
- [Co-teaching: Robust Training of Deep Neural Networks with Extremely Noisy Labels](https://mlsanity.com/paper/1804.06872/) (2018)
- [Unsupervised Label Noise Modeling and Loss Correction](https://mlsanity.com/paper/1904.11238/) (2019)
- [Augmentation Strategies for Learning with Noisy Labels](https://mlsanity.com/paper/2103.02130/) (2021)
- [Choosing the Sample with Lowest Loss makes SGD Robust](https://mlsanity.com/paper/2001.03316/) (2020)
- [Sarcasm Detection using Hybrid Neural Network](https://mlsanity.com/paper/1908.07414/) (2019)
- [Exponentiated Gradient Reweighting for Robust Training Under Label Noise and Beyond](https://mlsanity.com/paper/2104.01493/) (2021)
- [Towards Robust Learning with Different Label Noise Distributions](https://mlsanity.com/paper/1912.08741/) (2019)
- [Learning from Noisy Labels via Dynamic Loss Thresholding](https://mlsanity.com/paper/2104.02570/) (2021)
- [Cyclical Curriculum Learning](https://mlsanity.com/paper/2202.05531/) (2022)

## Abstract (arXiv)

SGD does not produce robust results on datasets with label noise. Because the gradients calculated according to the losses of the noisy samples cause the optimization process to go in the wrong direction. In this paper, as an alternative to SGD, we recommend using samples with loss less than a threshold value determined during the optimization process, instead of using all samples in the mini-batch. Our proposed method, Adaptive-k, aims to exclude label noise samples from the optimization process and make the process robust. On noisy datasets, we found that using a threshold-based approach, such as Adaptive-k, produces better results than using all samples or a fixed number of low-loss samples in the mini-batch. Based on our theoretical analysis and experimental results, we show that the Adaptive-k method is closest to the performance of the oracle, in which noisy samples are entirely removed from the dataset. Adaptive-k is a simple but effective method. It does not require prior knowledge of the noise ratio of the dataset, does not require additional model training, and does not increase training time significantly. The code for Adaptive-k is available at https://github.com/enesdedeoglu-TR/Adaptive-k