# Weights Augmentation: it has never ever ever ever let her model down

arXiv: https://arxiv.org/abs/2405.19590 · HTML: https://mlsanity.com/paper/2405.19590/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 29 of its 44 references that are in the map.
- First posted on arXiv 30 May 2024.

## Builds on (most cited first, 29 of 29)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Very Deep Convolutional Networks for Large-Scale Image Recognition](https://mlsanity.com/paper/1409.1556/) (2014)
- [EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks](https://mlsanity.com/paper/1905.11946/) (2019)
- [MobileNetV2: Inverted Residuals and Linear Bottlenecks](https://mlsanity.com/paper/1801.04381/) (2018)
- [Going Deeper with Convolutions](https://mlsanity.com/paper/1409.4842/) (2014)
- [RandAugment: Practical automated data augmentation with a reduced search space](https://mlsanity.com/paper/1909.13719/) (2019)
- [Knowledge Distillation: A Survey](https://mlsanity.com/paper/2006.05525/) (2020)
- [Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference time](https://mlsanity.com/paper/2203.05482/) (2022)
- [Averaging Weights Leads to Wider Optima and Better Generalization](https://mlsanity.com/paper/1803.05407/) (2018)
- [How Can We Know What Language Models Know?](https://mlsanity.com/paper/1911.12543/) (2019)
- [Robust fine-tuning of zero-shot models](https://mlsanity.com/paper/2109.01903/) (2021)
- [RepVGG: Making VGG-style ConvNets Great Again](https://mlsanity.com/paper/2101.03697/) (2021)
- [Be Your Own Teacher: Improve the Performance of Convolutional Neural Networks via Self Distillation](https://mlsanity.com/paper/1905.08094/) (2019)
- [Bag of Tricks for Image Classification with Convolutional Neural Networks](https://mlsanity.com/paper/1812.01187/) (2018)
- [BatchEnsemble: An Alternative Approach to Efficient Ensemble and Lifelong Learning](https://mlsanity.com/paper/2002.06715/) (2020)
- [Effective Data Augmentation With Diffusion Models](https://mlsanity.com/paper/2302.07944/) (2023)
- [Learning with Pseudo-Ensembles](https://mlsanity.com/paper/1412.4864/) (2014)
- [Model Fusion via Optimal Transport](https://mlsanity.com/paper/1910.05653/) (2019)
- [Adversarial Graph Augmentation to Improve Graph Contrastive Learning](https://mlsanity.com/paper/2106.05819/) (2021)
- [Text-to-Audio Generation using Instruction-Tuned LLM and Latent Diffusion Model](https://mlsanity.com/paper/2304.13731/) (2023)
- [Learning Data Augmentation Strategies for Object Detection](https://mlsanity.com/paper/1906.11172/) (2019)
- [Image Data Augmentation for Deep Learning: A Survey](https://mlsanity.com/paper/2204.08610/) (2022)
- [Stop Wasting My Time! Saving Days of ImageNet and BERT Training with Latest Weight Averaging](https://mlsanity.com/paper/2209.14981/) (2022)
- [Learning from models beyond fine-tuning](https://mlsanity.com/paper/2310.08184/) (2023)
- [What Makes a "Good" Data Augmentation in Knowledge Distillation -- A Statistical Perspective](https://mlsanity.com/paper/2012.02909/) (2020)
- [Improving Auto-Augment via Augmentation-Wise Weight Sharing](https://mlsanity.com/paper/2009.14737/) (2020)
- [Dropout as data augmentation](https://mlsanity.com/paper/1506.08700/) (2015)
- [Toward Understanding Generative Data Augmentation](https://mlsanity.com/paper/2305.17476/) (2023)

## Built on by (most cited first, 1 of 1)

- [Noisy Deep Ensemble: Accelerating Deep Ensemble Learning via Noise Injection](https://mlsanity.com/paper/2504.05677/) (2025)

## Abstract (arXiv)

Weight play an essential role in deep learning network models. Unlike network structure design, this article proposes the concept of weight augmentation, focusing on weight exploration. The core of Weight Augmentation Strategy (WAS) is to adopt random transformed weight coefficients training and transformed coefficients, named Shadow Weight(SW), for networks that can be used to calculate loss function to affect parameter updates. However, stochastic gradient descent is applied to Plain Weight(PW), which is referred to as the original weight of the network before the random transformation. During training, numerous SW collectively form high-dimensional space, while PW is directly learned from the distribution of SW instead of the data. The weight of the accuracy-oriented mode(AOM) relies on PW, which guarantees the network is highly robust and accurate. The desire-oriented mode(DOM) weight uses SW, which is determined by the network model's unique functions based on WAT's performance desires, such as lower computational complexity, lower sensitivity to particular data, etc. The dual mode be switched at anytime if needed. WAT extends the augmentation technique from data augmentation to weight, and it is easy to understand and implement, but it can improve almost all networks amazingly. Our experimental results show that convolutional neural networks, such as VGG-16, ResNet-18, ResNet-34, GoogleNet, MobilementV2, and Efficientment-Lite, can benefit much at little or no cost. The accuracy of models is on the CIFAR100 and CIFAR10 datasets, which can be evaluated to increase by 7.32% and 9.28%, respectively, with the highest values being 13.42% and 18.93%, respectively. In addition, DOM can reduce floating point operations (FLOPs) by up to 36.33%. The code is available at https://github.com/zlearh/Weight-Augmentation-Technology.