# Parameter Re-Initialization through Cyclical Batch Size Schedules

arXiv: https://arxiv.org/abs/1812.01216 · HTML: https://mlsanity.com/paper/1812.01216/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 4 Dec 2018.

## Built on by (most cited first, 3 of 3)

- [A Natural Bias for Language Generation Models](https://mlsanity.com/paper/2212.09686/) (2022)
- [AI Based Digital Twin Model for Cattle Caring](https://mlsanity.com/paper/2205.04034/) (2022)
- [Hyper-Learning for Gradient-Based Batch Size Adaptation](https://mlsanity.com/paper/2205.08231/) (2022)

## Abstract (arXiv)

Optimal parameter initialization remains a crucial problem for neural network training. A poor weight initialization may take longer to train and/or converge to sub-optimal solutions. Here, we propose a method of weight re-initialization by repeated annealing and injection of noise in the training process. We implement this through a cyclical batch size schedule motivated by a Bayesian perspective of neural network training. We evaluate our methods through extensive experiments on tasks in language modeling, natural language inference, and image classification. We demonstrate the ability of our method to improve language modeling performance by up to 7.91 perplexity and reduce training iterations by up to 61%, in addition to its flexibility in enabling snapshot ensembling and use with adversarial training.