# Do Deep Neural Networks Always Perform Better When Eating More Data?

arXiv: https://arxiv.org/abs/2205.15187 · HTML: https://mlsanity.com/paper/2205.15187/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 27 of its 57 references that are in the map.
- First posted on arXiv 30 May 2022.

## Builds on (most cited first, 27 of 27)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Very Deep Convolutional Networks for Large-Scale Image Recognition](https://mlsanity.com/paper/1409.1556/) (2014)
- [Grad-CAM: Visual Explanations from Deep Networks via Gradient-based Localization](https://mlsanity.com/paper/1610.02391/) (2016)
- [On the Opportunities and Risks of Foundation Models](https://mlsanity.com/paper/2108.07258/) (2021)
- [Wide Residual Networks](https://mlsanity.com/paper/1605.07146/) (2016)
- [Prototypical Networks for Few-shot Learning](https://mlsanity.com/paper/1703.05175/) (2017)
- [Matching Networks for One Shot Learning](https://mlsanity.com/paper/1606.04080/) (2016)
- [Shortcut Learning in Deep Neural Networks](https://mlsanity.com/paper/2004.07780/) (2020)
- [Revisiting Unreasonable Effectiveness of Data in Deep Learning Era](https://mlsanity.com/paper/1707.02968/) (2017)
- [Pseudo-LiDAR from Visual Depth Estimation: Bridging the Gap in 3D Object Detection for Autonomous Driving](https://mlsanity.com/paper/1812.07179/) (2018)
- [Towards Out-Of-Distribution Generalization: A Survey](https://mlsanity.com/paper/2108.13624/) (2021)
- [Multi-Modal Fusion Transformer for End-to-End Autonomous Driving](https://mlsanity.com/paper/2104.09224/) (2021)
- [ICNet for Real-Time Semantic Segmentation on High-Resolution Images](https://mlsanity.com/paper/1704.08545/) (2017)
- [FSDR: Frequency Space Domain Randomization for Domain Generalization](https://mlsanity.com/paper/2103.02370/) (2021)
- [Segmentation-Based Deep-Learning Approach for Surface-Defect Detection](https://mlsanity.com/paper/1903.08536/) (2019)
- [Deep Learning for LiDAR Point Clouds in Autonomous Driving: A Review](https://mlsanity.com/paper/2005.09830/) (2020)
- [Equalization Loss v2: A New Gradient Balance Approach for Long-tailed Object Detection](https://mlsanity.com/paper/2012.08548/) (2020)
- [Evidential Deep Learning for Open Set Action Recognition](https://mlsanity.com/paper/2107.10161/) (2021)
- [AI in Finance: Challenges, Techniques and Opportunities](https://mlsanity.com/paper/2107.09051/) (2021)
- [Semantic Segmentation for Real Point Cloud Scenes via Bilateral Augmentation and Adaptive Fusion](https://mlsanity.com/paper/2103.07074/) (2021)
- [Learning a Proposal Classifier for Multiple Object Tracking](https://mlsanity.com/paper/2103.07889/) (2021)
- [Semantic Concentration for Domain Adaptation](https://mlsanity.com/paper/2108.05720/) (2021)
- [Learning with Noisy Labels via Sparse Regularization](https://mlsanity.com/paper/2108.00192/) (2021)
- [PIT: Position-Invariant Transform for Cross-FoV Domain Adaptation](https://mlsanity.com/paper/2108.07142/) (2021)
- [CoMoGAN: continuous model-guided image-to-image translation](https://mlsanity.com/paper/2103.06879/) (2021)
- [Diverse Pretrained Context Encodings Improve Document Translation](https://mlsanity.com/paper/2106.03717/) (2021)
- [Different Set Domain Adaptation for Brain-Computer Interfaces: A Label Alignment Approach](https://mlsanity.com/paper/1912.01166/) (2019)

## Built on by (most cited first, 1 of 1)

- [Data-Centric Debugging: mitigating model failures via targeted data collection](https://mlsanity.com/paper/2211.09859/) (2022)

## Abstract (arXiv)

Data has now become a shortcoming of deep learning. Researchers in their own fields share the thinking that "deep neural networks might not always perform better when they eat more data," which still lacks experimental validation and a convincing guiding theory. Here to fill this lack, we design experiments from Identically Independent Distribution(IID) and Out of Distribution(OOD), which give powerful answers. For the purpose of guidance, based on the discussion of results, two theories are proposed: under IID condition, the amount of information determines the effectivity of each sample, the contribution of samples and difference between classes determine the amount of sample information and the amount of class information; under OOD condition, the cross-domain degree of samples determine the contributions, and the bias-fitting caused by irrelevant elements is a significant factor of cross-domain. The above theories provide guidance from the perspective of data, which can promote a wide range of practical applications of artificial intelligence.