# Improving Sample and Feature Selection with Principal Covariates Regression

arXiv: https://arxiv.org/abs/2012.12253 · HTML: https://mlsanity.com/paper/2012.12253/

## Key facts

- Cited by 9 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 22 Dec 2020.

## Built on by (most cited first, 9 of 9)

- [Characterizing metastable states with the help of machine learning](https://mlsanity.com/paper/2204.07391/) (2022)
- [Optimal radial basis for density-based atomic representations](https://mlsanity.com/paper/2105.08717/) (2021)
- [A smooth basis for atomistic machine learning](https://mlsanity.com/paper/2209.01948/) (2022)
- [On minimizing the training set fill distance in machine learning regression](https://mlsanity.com/paper/2307.10988/) (2023)
- [Feature Selection via Robust Weighted Score for High Dimensional Binary Class-Imbalanced Gene Expression Data](https://mlsanity.com/paper/2401.12667/) (2024)
- [Dislocation cartography: Representations and unsupervised classification of dislocation networks with unique fingerprints](https://mlsanity.com/paper/2406.15004/) (2024)
- [Mitigating Denial of Service Attacks in Fog-Based Wireless Sensor Networks Using Machine Learning Techniques](https://mlsanity.com/paper/2310.05952/) (2023)
- [Interpretable Visualizations of Data Spaces for Classification Problems](https://mlsanity.com/paper/2503.05861/) (2025)
- [Comparing the latent features of universal machine-learning interatomic potentials](https://mlsanity.com/paper/2512.05717/) (2025)

## Abstract (arXiv)

Selecting the most relevant features and samples out of a large set of candidates is a task that occurs very often in the context of automated data analysis, where it can be used to improve the computational performance, and also often the transferability, of a model. Here we focus on two popular sub-selection schemes which have been applied to this end: CUR decomposition, that is based on a low-rank approximation of the feature matrix and Farthest Point Sampling, that relies on the iterative identification of the most diverse samples and discriminating features. We modify these unsupervised approaches, incorporating a supervised component following the same spirit as the Principal Covariates Regression (PCovR) method. We show that incorporating target information provides selections that perform better in supervised tasks, which we demonstrate with ridge regression, kernel ridge regression, and sparse kernel regression. We also show that incorporating aspects of simple supervised learning models can improve the accuracy of more complex models, such as feed-forward neural networks. We present adjustments to minimize the impact that any subselection may incur when performing unsupervised tasks. We demonstrate the significant improvements associated with the use of PCov-CUR and PCov-FPS selections for applications to chemistry and materials science, typically reducing by a factor of two the number of features and samples which are required to achieve a given level of regression accuracy.