# PCA-Initialized Deep Neural Networks Applied To Document Image Analysis

arXiv: https://arxiv.org/abs/1702.00177 · HTML: https://mlsanity.com/paper/1702.00177/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 1 Feb 2017.

## Built on by (most cited first, 6 of 6)

- [EAML: Ensemble Self-Attention-based Mutual Learning Network for Document Image Classification](https://mlsanity.com/paper/2305.06923/) (2023)
- [E-LMC: Extended Linear Model of Coregionalization for Spatial Field Prediction](https://mlsanity.com/paper/2203.00525/) (2022)
- [PCA-Boosted Autoencoders for Nonlinear Dimensionality Reduction in Low Data Regimes](https://mlsanity.com/paper/2205.11673/) (2022)
- [Mirror descent of Hopfield model](https://mlsanity.com/paper/2211.15880/) (2022)
- [Deep Symmetric Autoencoders from the Eckart-Young-Schmidt Perspective](https://mlsanity.com/paper/2506.11641/) (2025)
- [Designing the Architecture of a Convolutional Neural Network Automatically for Diabetic Retinopathy Diagnosis](https://mlsanity.com/paper/2110.03877/) (2021)

## Abstract (arXiv)

In this paper, we present a novel approach for initializing deep neural networks, i.e., by turning PCA into neural layers. Usually, the initialization of the weights of a deep neural network is done in one of the three following ways: 1) with random values, 2) layer-wise, usually as Deep Belief Network or as auto-encoder, and 3) re-use of layers from another network (transfer learning). Therefore, typically, many training epochs are needed before meaningful weights are learned, or a rather similar dataset is required for seeding a fine-tuning of transfer learning. In this paper, we describe how to turn a PCA into an auto-encoder, by generating an encoder layer of the PCA parameters and furthermore adding a decoding layer. We analyze the initialization technique on real documents. First, we show that a PCA-based initialization is quick and leads to a very stable initialization. Furthermore, for the task of layout analysis we investigate the effectiveness of PCA-based initialization and show that it outperforms state-of-the-art random weight initialization methods.