# Principal Components for Neural Network Initialization

arXiv: https://arxiv.org/abs/2501.19114 · HTML: https://mlsanity.com/paper/2501.19114/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 8 of its 21 references that are in the map.
- First posted on arXiv 31 Jan 2025.

## Builds on (most cited first, 8 of 8)

- [Grad-CAM: Visual Explanations from Deep Networks via Gradient-based Localization](https://mlsanity.com/paper/1610.02391/) (2016)
- [A Unified Approach to Interpreting Model Predictions](https://mlsanity.com/paper/1705.07874/) (2017)
- ["Why Should I Trust You?": Explaining the Predictions of Any Classifier](https://mlsanity.com/paper/1602.04938/) (2016)
- [Visualizing and Understanding Convolutional Networks](https://mlsanity.com/paper/1311.2901/) (2013)
- [All you need is a good init](https://mlsanity.com/paper/1511.06422/) (2015)
- [Multiple imputation for continuous variables using a Bayesian principal component analysis](https://mlsanity.com/paper/1401.5747/) (2014)
- [From neural PCA to deep unsupervised learning](https://mlsanity.com/paper/1411.7783/) (2014)
- [Blockwise Principal Component Analysis for monotone missing data imputation and dimensionality reduction](https://mlsanity.com/paper/2305.06042/) (2023)

## Built on by (most cited first, 2 of 2)

- [Dimensional Collapse in Transformer Attention Outputs: A Challenge for Sparse Dictionary Learning](https://mlsanity.com/paper/2508.16929/) (2025)
- [Prior-Informed Neural Network Initialization: A Spectral Approach for Function Parameterizing Architectures](https://mlsanity.com/paper/2603.16376/) (2026)

## Abstract (arXiv)

Principal Component Analysis (PCA) is a commonly used tool for dimension reduction and denoising. Therefore, it is also widely used on the data prior to training a neural network. However, this approach can complicate the explanation of eXplainable Artificial Intelligence (XAI) methods for the decision of the model. In this work, we analyze the potential issues with this approach and propose Principal Components-based Initialization (PCsInit), a strategy to incorporate PCA into the first layer of a neural network via initialization of the first layer in the network with the principal components, and its two variants PCsInit-Act and PCsInit-Sub. We will show that explanations using these strategies are more simple, direct and straightforward than using PCA prior to training a neural network on the principal components. We also show that the proposed techniques possess desirable theoretical properties. Moreover, as will be illustrated in the experiments, such training strategies can also allow further improvement of training via backpropagation compared to training neural networks on principal components.