Neural Network Characterization and Entropy Regulated Data Balancing through Principal Component Analysis

This paper examines in detail the geometric structure of principal component analysis (PCA) by considering in detail the distributions of both unrotated and rotated MNIST digits in the space defined by the lowest order PCA components. Since digits possessing salient geometric features are mapped to restricted regions far from the origin, they are predicted by neural networks with a greater accuracy than digits that are mapped to broad, diffuse and overlapping volumes of the low order PCA space. Motivated by these results, a new quantity, the local PCA entropy, obtained by dividing the spatial region spanned by the low order principal components into histogram bins and evaluating the entropy associated with the number of occurrences of each input class within a bin, is introduced. The metric locates the input data records that yield the largest confusion in prediction accuracy within reduced coordinate volumes that optimally discriminate among geometric features. As an example of the potential utility of the local PCA entropy, a simple data balancing procedure is realized by oversampling the data records in regions of large local entropy.