# Weight Update Skipping: Reducing Training Time for Artificial Neural Networks

arXiv: https://arxiv.org/abs/2012.02792 · HTML: https://mlsanity.com/paper/2012.02792/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 5 Dec 2020.

## Built on by (most cited first, 6 of 6)

- [IISAN: Efficiently Adapting Multimodal Representation for Sequential Recommendation with Decoupled PEFT](https://mlsanity.com/paper/2404.02059/) (2024)
- [Layer Freezing & Data Sieving: Missing Pieces of a Generic Framework for Sparse Training](https://mlsanity.com/paper/2209.11204/) (2022)
- [Enhancing Deep Neural Network Training Efficiency and Performance through Linear Prediction](https://mlsanity.com/paper/2310.10958/) (2023)
- [Kernel Normalized Convolutional Networks](https://mlsanity.com/paper/2205.10089/) (2022)
- [Early Exiting Predictive Coding Neural Networks for Edge AI](https://mlsanity.com/paper/2309.02022/) (2023)
- [Tri-Accel: Curvature-Aware Precision-Adaptive and Memory-Elastic Optimization for Efficient GPU Usage](https://mlsanity.com/paper/2508.16905/) (2025)

## Abstract (arXiv)

Artificial Neural Networks (ANNs) are known as state-of-the-art techniques in Machine Learning (ML) and have achieved outstanding results in data-intensive applications, such as recognition, classification, and segmentation. These networks mostly use deep layers of convolution or fully connected layers with many filters in each layer, demanding a large amount of data and tunable hyperparameters to achieve competitive accuracy. As a result, storage, communication, and computational costs of training (in particular training time) become limiting factors to scale them up. In this paper, we propose a new training methodology for ANNs that exploits the observation of improvement of accuracy shows temporal variations which allow us to skip updating weights when the variation is minuscule. During such time windows, we keep updating bias which ensures the network still trains and avoids overfitting; however, we selectively skip updating weights (and their time-consuming computations). Such a training approach virtually achieves the same accuracy with considerably less computational cost, thus lower training time. We propose two methods for updating weights and evaluate them by analyzing four state-of-the-art models, AlexNet, VGG-11, VGG-16, ResNet-18 on CIFAR datasets. On average, our two proposed methods called WUS and WUS+LR reduced the training time (compared to the baseline) by 54%, and 50%, respectively on CIFAR-10; and 43% and 35% on CIFAR-100, respectively.