# How does Weight Correlation Affect the Generalisation Ability of Deep Neural Networks

arXiv: https://arxiv.org/abs/2010.05983 · HTML: https://mlsanity.com/paper/2010.05983/

## Key facts

- Cited by 32 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 12 Oct 2020.

## Built on by (most cited first, 30 of 32)

- [Enhancing Adversarial Training with Second-Order Statistics of Weights](https://mlsanity.com/paper/2203.06020/) (2022)
- [TANGOS: Regularizing Tabular Neural Networks through Gradient Orthogonalization and Specialization](https://mlsanity.com/paper/2303.05506/) (2023)
- [Randomized Adversarial Training via Taylor Expansion](https://mlsanity.com/paper/2303.10653/) (2023)
- [Improving GANs for Long-Tailed Data through Group Spectral Regularization](https://mlsanity.com/paper/2208.09932/) (2022)
- [Towards Green AI in Fine-tuning Large Language Models via Adaptive Backpropagation](https://mlsanity.com/paper/2309.13192/) (2023)
- [Enhancing Robust Fairness via Confusional Spectral Regularization](https://mlsanity.com/paper/2501.13273/) (2025)
- [Weight Expansion: A New Perspective on Dropout and Generalization](https://mlsanity.com/paper/2201.09209/) (2022)
- [Adversarial Robustness of Deep Learning: Theory, Algorithms, and Applications](https://mlsanity.com/paper/2108.10451/) (2021)
- [3DVerifier: Efficient Robustness Verification for 3D Point Cloud Models](https://mlsanity.com/paper/2207.07539/) (2022)
- [Towards A Unified PAC-Bayesian Framework for Norm-based Generalization Bounds](https://mlsanity.com/paper/2601.08100/) (2026)
- [Connecting Independently Trained Modes via Layer-Wise Connectivity](https://mlsanity.com/paper/2505.02604/) (2025)
- [On the Stability of the Jacobian Matrix in Deep Neural Networks](https://mlsanity.com/paper/2506.08764/) (2025)
- [Neuronal Correlation: a Central Concept in Neural Network](https://mlsanity.com/paper/2201.09069/) (2022)
- [Shapley Uncertainty in Natural Language Generation](https://mlsanity.com/paper/2507.21406/) (2025)
- [Reconcile Certified Robustness and Accuracy for DNN-based Smoothed Majority Vote Classifier](https://mlsanity.com/paper/2509.25979/) (2025)
- [Abstraction and Symbolic Execution of Deep Neural Networks with Bayesian Approximation of Hidden Features](https://mlsanity.com/paper/2103.03704/) (2021)
- [On the Adversarial Vulnerabilities of Transfer Learning in Remote Sensing](https://mlsanity.com/paper/2501.11462/) (2025)
- [Information Consistent Pruning: How to Efficiently Search for Sparse Networks?](https://mlsanity.com/paper/2501.15592/) (2025)
- [KO: Kinetics-inspired Neural Optimizer with PDE Simulation Approaches](https://mlsanity.com/paper/2505.14777/) (2025)
- [How Weight Resampling and Optimizers Shape the Dynamics of Continual Learning and Forgetting in Neural Networks](https://mlsanity.com/paper/2507.01559/) (2025)
- [Margin-Adaptive Confidence Ranking for Reliable LLM Judgement](https://mlsanity.com/paper/2605.15416/) (2026)
- [Localize-Then-Decide Guarantees for LLM Judgments](https://mlsanity.com/paper/2608.25824/) (2026)
- [Using Intermediate Forward Iterates for Intermediate Generator Optimization](https://mlsanity.com/paper/2302.02336/) (2023)
- [A Dataset Fusion Algorithm for Generalised Anomaly Detection in Homogeneous Periodic Time Series Datasets](https://mlsanity.com/paper/2305.08197/) (2023)
- [Understanding Activation Patterns in Artificial Neural Networks by Exploring Stochastic Processes](https://mlsanity.com/paper/2308.00858/) (2023)
- [Decorrelating neurons using persistence](https://mlsanity.com/paper/2308.04870/) (2023)
- [Invariant Correlation of Representation with Label: Enhancing Domain Generalization in Noisy Environments](https://mlsanity.com/paper/2407.01749/) (2024)
- [Principal Eigenvalue Regularization for Improved Worst-Class Certified Robustness of Smoothed Classifiers](https://mlsanity.com/paper/2503.17172/) (2025)
- [Hyperflux: Pruning Reveals Importance](https://mlsanity.com/paper/2504.05349/) (2025)
- [Emergence of Structure in Ensembles of Random Neural Networks](https://mlsanity.com/paper/2505.10331/) (2025)

## Abstract (arXiv)

This paper studies the novel concept of weight correlation in deep neural networks and discusses its impact on the networks' generalisation ability. For fully-connected layers, the weight correlation is defined as the average cosine similarity between weight vectors of neurons, and for convolutional layers, the weight correlation is defined as the cosine similarity between filter matrices. Theoretically, we show that, weight correlation can, and should, be incorporated into the PAC Bayesian framework for the generalisation of neural networks, and the resulting generalisation bound is monotonic with respect to the weight correlation. We formulate a new complexity measure, which lifts the PAC Bayes measure with weight correlation, and experimentally confirm that it is able to rank the generalisation errors of a set of networks more precisely than existing measures. More importantly, we develop a new regulariser for training, and provide extensive experiments that show that the generalisation error can be greatly reduced with our novel approach.