# Improving Data Quality with Training Dynamics of Gradient Boosting Decision Trees

arXiv: https://arxiv.org/abs/2210.11327 · HTML: https://mlsanity.com/paper/2210.11327/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 19 of its 49 references that are in the map.
- First posted on arXiv 20 Oct 2022.

## Builds on (most cited first, 19 of 19)

- [XGBoost: A Scalable Tree Boosting System](https://mlsanity.com/paper/1603.02754/) (2016)
- [Optuna: A Next-generation Hyperparameter Optimization Framework](https://mlsanity.com/paper/1907.10902/) (2019)
- [An Empirical Study of Example Forgetting during Deep Neural Network Learning](https://mlsanity.com/paper/1812.05159/) (2018)
- [Tabular Data: Deep Learning is Not All You Need](https://mlsanity.com/paper/2106.03253/) (2021)
- [Beyond neural scaling laws: beating power law scaling via data pruning](https://mlsanity.com/paper/2206.14486/) (2022)
- [Does Learning Require Memorization? A Short Tale about a Long Tail](https://mlsanity.com/paper/1906.05271/) (2019)
- [Dataset Cartography: Mapping and Diagnosing Datasets with Training Dynamics](https://mlsanity.com/paper/2009.10795/) (2020)
- [Early-Learning Regularization Prevents Memorization of Noisy Labels](https://mlsanity.com/paper/2007.00151/) (2020)
- [Confident Learning: Estimating Uncertainty in Dataset Labels](https://mlsanity.com/paper/1911.00068/) (2019)
- [Identifying Mislabeled Data using the Area Under the Margin Ranking](https://mlsanity.com/paper/2001.10528/) (2020)
- [Understanding and Utilizing Deep Neural Networks Trained with Noisy Labels](https://mlsanity.com/paper/1905.05040/) (2019)
- [REPAIR: Removing Representation Bias by Dataset Resampling](https://mlsanity.com/paper/1904.07911/) (2019)
- [How Complex is your classification problem? A survey on measuring classification complexity](https://mlsanity.com/paper/1808.03591/) (2018)
- [Loss factorization, weakly supervised learning and label noise robustness](https://mlsanity.com/paper/1602.02450/) (2016)
- [Are All Training Examples Created Equal? An Empirical Study](https://mlsanity.com/paper/1811.12569/) (2018)
- [Double Ramp Loss Based Reject Option Classifier](https://mlsanity.com/paper/1311.6556/) (2013)
- [Assessing the Quality of the Datasets by Identifying Mislabeled Samples](https://mlsanity.com/paper/2109.05000/) (2021)
- [Training Deep Networks from Zero to Hero: avoiding pitfalls and going beyond](https://mlsanity.com/paper/2109.02752/) (2021)
- [Automatic Feasibility Study via Data Quality Analysis for ML: A Case-Study on Label Noise](https://mlsanity.com/paper/2010.08410/) (2020)

## Built on by (most cited first, 3 of 3)

- [Decoupling Decision-Making in Fraud Prevention through Classifier Calibration for Business Logic Action](https://mlsanity.com/paper/2401.05240/) (2024)
- [Training Gradient Boosted Decision Trees on Tabular Data Containing Label Noise for Classification Tasks](https://mlsanity.com/paper/2409.08647/) (2024)
- [Selfish Evolution: Making Discoveries in Extreme Label Noise with the Help of Overfitting Dynamics](https://mlsanity.com/paper/2412.00077/) (2024)

## Abstract (arXiv)

Real world datasets contain incorrectly labeled instances that hamper the performance of the model and, in particular, the ability to generalize out of distribution. Also, each example might have different contribution towards learning. This motivates studies to better understanding of the role of data instances with respect to their contribution in good metrics in models. In this paper we propose a method based on metrics computed from training dynamics of Gradient Boosting Decision Trees (GBDTs) to assess the behavior of each training example. We focus on datasets containing mostly tabular or structured data, for which the use of Decision Trees ensembles are still the state-of-the-art in terms of performance. Our methods achieved the best results overall when compared with confident learning, direct heuristics and a robust boosting algorithm. We show results on detecting noisy labels in order clean datasets, improving models' metrics in synthetic and real public datasets, as well as on a industry case in which we deployed a model based on the proposed solution.