# On Data-centric Myths

arXiv: https://arxiv.org/abs/2111.11514 · HTML: https://mlsanity.com/paper/2111.11514/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 15 of its 19 references that are in the map.
- First posted on arXiv 22 Nov 2021.

## Builds on (most cited first, 15 of 15)

- [Very Deep Convolutional Networks for Large-Scale Image Recognition](https://mlsanity.com/paper/1409.1556/) (2014)
- [mixup: Beyond Empirical Risk Minimization](https://mlsanity.com/paper/1710.09412/) (2017)
- [Identity Mappings in Deep Residual Networks](https://mlsanity.com/paper/1603.05027/) (2016)
- [Domain Generalization by Solving Jigsaw Puzzles](https://mlsanity.com/paper/1903.06864/) (2019)
- [The Pitfalls of Simplicity Bias in Neural Networks](https://mlsanity.com/paper/2006.07710/) (2020)
- [Intrinsic dimension of data representations in deep neural networks](https://mlsanity.com/paper/1905.12784/) (2019)
- [Estimating the intrinsic dimension of datasets by a minimal neighborhood information](https://mlsanity.com/paper/1803.06992/) (2018)
- [Deep learning generalizes because the parameter-function map is biased towards simple functions](https://mlsanity.com/paper/1805.08522/) (2018)
- [What shapes feature representations? Exploring datasets, architectures, and training](https://mlsanity.com/paper/2006.12433/) (2020)
- [Data Augmentation by Pairing Samples for Images Classification](https://mlsanity.com/paper/1801.02929/) (2018)
- [On Mixup Regularization](https://mlsanity.com/paper/2006.06049/) (2020)
- [Affinity and Diversity: Quantifying Mechanisms of Data Augmentation](https://mlsanity.com/paper/2002.08973/) (2020)
- [FMix: Enhancing Mixed Sample Data Augmentation](https://mlsanity.com/paper/2002.12047/) (2020)
- [Distributional Results for Model-Based Intrinsic Dimension Estimators](https://mlsanity.com/paper/2104.13832/) (2021)
- [Bayesian Distance Clustering](https://mlsanity.com/paper/1810.08537/) (2018)

## Built on by (most cited first, 4 of 4)

- [Is one annotation enough? A data-centric image classification benchmark for noisy and ambiguous label estimation](https://mlsanity.com/paper/2207.06214/) (2022)
- [Beyond Hard Labels: Investigating data label distributions](https://mlsanity.com/paper/2207.06224/) (2022)
- [Stochastic Perturbations of Tabular Features for Non-Deterministic Inference with Automunge](https://mlsanity.com/paper/2202.09248/) (2022)
- [Geometric Regularization from Overparameterization](https://mlsanity.com/paper/2202.09276/) (2022)

## Abstract (arXiv)

The community lacks theory-informed guidelines for building good data sets. We analyse theoretical directions relating to what aspects of the data matter and conclude that the intuitions derived from the existing literature are incorrect and misleading. Using empirical counter-examples, we show that 1) data dimension should not necessarily be minimised and 2) when manipulating data, preserving the distribution is inessential. This calls for a more data-aware theoretical understanding. Although not explored in this work, we propose the study of the impact of data modification on learned representations as a promising research direction.