# PEP: Parameter Ensembling by Perturbation

arXiv: https://arxiv.org/abs/2010.12721 · HTML: https://mlsanity.com/paper/2010.12721/

## Key facts

- Cited by 9 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 24 Oct 2020.

## Built on by (most cited first, 9 of 9)

- [Triggering Failures: Out-Of-Distribution detection by learning from local adversarial attacks in Semantic Segmentation](https://mlsanity.com/paper/2108.01634/) (2021)
- [Neural Thickets: Diverse Task Experts Are Dense Around Pretrained Weights](https://mlsanity.com/paper/2603.12228/) (2026)
- [Sparse Model Soups: A Recipe for Improved Pruning via Model Averaging](https://mlsanity.com/paper/2306.16788/) (2023)
- [Likelihood Annealing: Fast Calibrated Uncertainty for Regression](https://mlsanity.com/paper/2302.11012/) (2023)
- [Sharpness-diversity tradeoff: improving flat ensembles with SharpBalance](https://mlsanity.com/paper/2407.12996/) (2024)
- [Uncertainty Quantification With Noise Injection in Neural Networks: A Bayesian Perspective](https://mlsanity.com/paper/2501.12314/) (2025)
- [Not the Dimension, the Norm: What Matters in Gradient-Free Weight Perturbation of Language Models](https://mlsanity.com/paper/2608.01624/) (2026)
- [Instance-Aware Observer Network for Out-of-Distribution Object Segmentation](https://mlsanity.com/paper/2207.08782/) (2022)
- [Modular Norm RandOpt: Population-Efficient Ensembling through Architecture-Aware Perturbations](https://mlsanity.com/paper/2609.25745/) (2026)

## Abstract (arXiv)

Ensembling is now recognized as an effective approach for increasing the predictive performance and calibration of deep networks. We introduce a new approach, Parameter Ensembling by Perturbation (PEP), that constructs an ensemble of parameter values as random perturbations of the optimal parameter set from training by a Gaussian with a single variance parameter. The variance is chosen to maximize the log-likelihood of the ensemble average (mathbbL) on the validation data set. Empirically, and perhaps surprisingly, mathbbL has a well-defined maximum as the variance grows from zero (which corresponds to the baseline model). Conveniently, calibration level of predictions also tends to grow favorably until the peak of mathbbL is reached. In most experiments, PEP provides a small improvement in performance, and, in some cases, a substantial improvement in empirical calibration. We show that this "PEP effect" (the gain in log-likelihood) is related to the mean curvature of the likelihood function and the empirical Fisher information. Experiments on ImageNet pre-trained networks including ResNet, DenseNet, and Inception showed improved calibration and likelihood. We further observed a mild improvement in classification accuracy on these networks. Experiments on classification benchmarks such as MNIST and CIFAR-10 showed improved calibration and likelihood, as well as the relationship between the PEP effect and overfitting; this demonstrates that PEP can be used to probe the level of overfitting that occurred during training. In general, no special training procedure or network architecture is needed, and in the case of pre-trained networks, no additional training is needed.