# Predicting with High Correlation Features

arXiv: https://arxiv.org/abs/1910.00164 · HTML: https://mlsanity.com/paper/1910.00164/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 1 Oct 2019.

## Built on by (most cited first, 5 of 5)

- [OOD-GNN: Out-of-Distribution Generalized Graph Neural Network](https://mlsanity.com/paper/2112.03806/) (2021)
- [Out-of-distribution Generalization with Causal Invariant Transformations](https://mlsanity.com/paper/2203.11528/) (2022)
- [Breaking Correlation Shift via Conditional Invariant Regularizer](https://mlsanity.com/paper/2207.06687/) (2022)
- [Gradient Masked Federated Optimization](https://mlsanity.com/paper/2104.10322/) (2021)
- [Domain Generalization via Selective Consistency Regularization for Time Series Classification](https://mlsanity.com/paper/2206.07876/) (2022)

## Abstract (arXiv)

It has been shown that instead of learning actual object features, deep networks tend to exploit non-robust (spurious) discriminative features that are shared between training and test sets. Therefore, while they achieve state of the art performance on such test sets, they achieve poor generalization on out of distribution (OOD) samples where the IID (independent, identical distribution) assumption breaks and the distribution of non-robust features shifts. In this paper, we consider distribution shift as a shift in the distribution of input features during test time that exhibit low correlation with targets in the training set. Under this definition, we evaluate existing robust feature learning methods and regularization methods and compare them against a baseline designed to specifically capture high correlation features in training set. As a controlled test-bed, we design a colored MNIST (C-MNIST) dataset and find that existing methods trained on this set fail to generalize well on an OOD version this dataset, showing that they overfit the low correlation color features. This is avoided by the baseline method trained on the same C-MNIST data, which is designed to learn high correlation features, and is able to generalize on the test sets of vanilla MNIST, MNIST-M and SVHN datasets. Our code is available at urlhttps://github.com/salesforce/corrbasedprediction.