# The Foes of Neural Network's Data Efficiency Among Unnecessary Input Dimensions

arXiv: https://arxiv.org/abs/2107.06409 · HTML: https://mlsanity.com/paper/2107.06409/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 14 of its 25 references that are in the map.
- First posted on arXiv 13 Jul 2021.

## Builds on (most cited first, 14 of 14)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Understanding deep learning requires rethinking generalization](https://mlsanity.com/paper/1611.03530/) (2016)
- [Reconciling modern machine learning practice and the bias-variance trade-off](https://mlsanity.com/paper/1812.11118/) (2018)
- [Deep Double Descent: Where Bigger Models and More Data Hurt](https://mlsanity.com/paper/1912.02292/) (2019)
- [Recognition in Terra Incognita](https://mlsanity.com/paper/1807.04975/) (2018)
- [Noise or Signal: The Role of Image Backgrounds in Object Recognition](https://mlsanity.com/paper/2006.09994/) (2020)
- [Intrinsic dimension of data representations in deep neural networks](https://mlsanity.com/paper/1905.12784/) (2019)
- [High-dimensional dynamics of generalization error in neural networks](https://mlsanity.com/paper/1710.03667/) (2017)
- [Object Recognition with and without Objects](https://mlsanity.com/paper/1611.06596/) (2016)
- [Dimensionality compression and expansion in Deep Neural Networks](https://mlsanity.com/paper/1906.00443/) (2019)
- [Foveation-based Mechanisms Alleviate Adversarial Examples](https://mlsanity.com/paper/1511.06292/) (2015)
- [Object Detection Through Exploration With A Foveated Visual Field](https://mlsanity.com/paper/1408.0814/) (2014)
- [Do Deep Neural Networks Suffer from Crowding?](https://mlsanity.com/paper/1706.08616/) (2017)
- [Contemplating real-world object classification](https://mlsanity.com/paper/2103.05137/) (2021)

## Built on by (most cited first, 1 of 1)

- [Relating Regularization and Generalization through the Intrinsic Dimension of Activations](https://mlsanity.com/paper/2211.13239/) (2022)

## Abstract (arXiv)

Datasets often contain input dimensions that are unnecessary to predict the output label, e.g. background in object recognition, which lead to more trainable parameters. Deep Neural Networks (DNNs) are robust to increasing the number of parameters in the hidden layers, but it is unclear whether this holds true for the input layer. In this letter, we investigate the impact of unnecessary input dimensions on a central issue of DNNs: their data efficiency, ie. the amount of examples needed to achieve certain generalization performance. Our results show that unnecessary input dimensions that are task-unrelated substantially degrade data efficiency. This highlights the need for mechanisms that remove {task-unrelated} dimensions to enable data efficiency gains.