# Fine Tuning with Abnormal Examples

arXiv: https://arxiv.org/abs/2304.13783 · HTML: https://mlsanity.com/paper/2304.13783/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 2 of its 4 references that are in the map.
- First posted on arXiv 26 Apr 2023.

## Builds on (most cited first, 2 of 2)

- [SQuAD: 100,000+ Questions for Machine Comprehension of Text](https://mlsanity.com/paper/1606.05250/) (2016)
- [Dataset Cartography: Mapping and Diagnosing Datasets with Training Dynamics](https://mlsanity.com/paper/2009.10795/) (2020)

## Abstract (arXiv)

Given the prevalence of crowd sourced labor in creating Natural Language processing datasets, these aforementioned sets have become increasingly large. For instance, the SQUAD dataset currently sits at over 80,000 records. However, because the English language is rather repetitive in structure, the distribution of word frequencies in the SQUAD dataset's contexts are relatively unchanged. By measuring each sentences distance from the co-variate distance of frequencies of all sentences in the dataset, we identify 10,500 examples that create a more uniform distribution for training. While fine-tuning ELECTRA [4] on this subset of examples reaches better performance to a model trained on all 87,000 examples. Herein we introduce a methodology for systematically pruning datasets for fine tuning reaching better out of sample performance.