# A General Multiple Data Augmentation Based Framework for Training Deep Neural Networks

arXiv: https://arxiv.org/abs/2205.14606 · HTML: https://mlsanity.com/paper/2205.14606/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 23 of its 38 references that are in the map.
- First posted on arXiv 29 May 2022.

## Builds on (most cited first, 23 of 23)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Distilling the Knowledge in a Neural Network](https://mlsanity.com/paper/1503.02531/) (2015)
- [Densely Connected Convolutional Networks](https://mlsanity.com/paper/1608.06993/) (2016)
- [mixup: Beyond Empirical Risk Minimization](https://mlsanity.com/paper/1710.09412/) (2017)
- [Wide Residual Networks](https://mlsanity.com/paper/1605.07146/) (2016)
- [CutMix: Regularization Strategy to Train Strong Classifiers with Localizable Features](https://mlsanity.com/paper/1905.04899/) (2019)
- [Benchmarking Neural Network Robustness to Common Corruptions and Perturbations](https://mlsanity.com/paper/1903.12261/) (2019)
- [Aggregated Residual Transformations for Deep Neural Networks](https://mlsanity.com/paper/1611.05431/) (2016)
- [RandAugment: Practical automated data augmentation with a reduced search space](https://mlsanity.com/paper/1909.13719/) (2019)
- [Improved Regularization of Convolutional Neural Networks with Cutout](https://mlsanity.com/paper/1708.04552/) (2017)
- [Born Again Neural Networks](https://mlsanity.com/paper/1805.04770/) (2018)
- [Be Your Own Teacher: Improve the Performance of Convolutional Neural Networks via Self Distillation](https://mlsanity.com/paper/1905.08094/) (2019)
- [Bag of Tricks for Image Classification with Convolutional Neural Networks](https://mlsanity.com/paper/1812.01187/) (2018)
- [ResNet strikes back: An improved training procedure in timm](https://mlsanity.com/paper/2110.00476/) (2021)
- [Re-labeling ImageNet: from Single to Multi-Labels, from Global to Localized Labels](https://mlsanity.com/paper/2101.05022/) (2021)
- [Convolutional Networks with Dense Connectivity](https://mlsanity.com/paper/2001.02394/) (2020)
- [Explaining Knowledge Distillation by Quantifying the Knowledge](https://mlsanity.com/paper/2003.03622/) (2020)
- [Label Refinery: Improving ImageNet Classification through Label Progression](https://mlsanity.com/paper/1805.02641/) (2018)
- [Peer Collaborative Learning for Online Knowledge Distillation](https://mlsanity.com/paper/2006.04147/) (2020)
- [Deep Model Compression: Distilling Knowledge from Noisy Teachers](https://mlsanity.com/paper/1610.09650/) (2016)
- [Circumventing Outliers of AutoAugment with Knowledge Distillation](https://mlsanity.com/paper/2003.11342/) (2020)
- [An overview of mixing augmentation methods and augmentation strategies](https://mlsanity.com/paper/2107.09887/) (2021)
- [Multi-view Contrastive Learning for Online Knowledge Distillation](https://mlsanity.com/paper/2006.04093/) (2020)

## Abstract (arXiv)

Deep neural networks (DNNs) often rely on massive labelled data for training, which is inaccessible in many applications. Data augmentation (DA) tackles data scarcity by creating new labelled data from available ones. Different DA methods have different mechanisms and therefore using their generated labelled data for DNN training may help improving DNN's generalisation to different degrees. Combining multiple DA methods, namely multi-DA, for DNN training, provides a way to boost generalisation. Among existing multi-DA based DNN training methods, those relying on knowledge distillation (KD) have received great attention. They leverage knowledge transfer to utilise the labelled data sets created by multiple DA methods instead of directly combining them for training DNNs. However, existing KD-based methods can only utilise certain types of DA methods, incapable of utilising the advantages of arbitrary DA methods. We propose a general multi-DA based DNN training framework capable to use arbitrary DA methods. To train a DNN, our framework replicates a certain portion in the latter part of the DNN into multiple copies, leading to multiple DNNs with shared blocks in their former parts and independent blocks in their latter parts. Each of these DNNs is associated with a unique DA and a newly devised loss that allows comprehensively learning from the data generated by all DA methods and the outputs from all DNNs in an online and adaptive way. The overall loss, i.e., the sum of each DNN's loss, is used for training the DNN. Eventually, one of the DNNs with the best validation performance is chosen for inference. We implement the proposed framework by using three distinct DA methods and apply it for training representative DNNs. Experiments on the popular benchmarks of image classification demonstrate the superiority of our method to several existing single-DA and multi-DA based training methods.