A New Covariance Estimator for Sufficient Dimension Reduction in High-Dimensional and Undersized Sample Problems

The application of standard sufficient dimension reduction methods for reducing the dimension space of predictors without losing regression information requires inverting the covariance matrix of the predictors. This has posed a number of challenges especially when analyzing high-dimensional data sets in which the number of predictors mathitp is much larger than number of samples n,~(nll p). A new covariance estimator, called the textitMaximum Entropy Covariance (MEC) that addresses loss of covariance information when similar covariance matrices are linearly combined using textitMaximum Entropy (ME) principle is proposed in this work. By benefitting naturally from slicing or discretizing range of the response variable, y into textitH non-overlapping categories, mathith1,ldots ,hH, MEC first combines covariance matrices arising from samples in each y slice mathith∈ H and then select the one that maximizes entropy under the principle of maximum uncertainty. The MEC estimator is then formed from convex mixture of such entropy-maximizing sample covariance S_mboxmec estimate and pooled sample covariance S_mathitp estimate across the mathitH slices without requiring time-consuming covariance optimization procedures. MEC deals directly with singularity and instability of sample group covariance estimate in both regression and classification problems. The efficiency of the MEC estimator is studied with the existing sufficient dimension reduction methods such as textitSliced Inverse Regression (SIR) and textitSliced Average Variance Estimator (SAVE) as demonstrated on both classification and regression problems using real life Leukemia cancer data and customers' electricity load profiles from smart meter data sets respectively.