# Combining Diverse Feature Priors

arXiv: https://arxiv.org/abs/2110.08220 · HTML: https://mlsanity.com/paper/2110.08220/

## Key facts

- Cited by 12 papers in the map (data through 14 Oct 2026).
- Builds on 56 of its 90 references that are in the map.
- First posted on arXiv 15 Oct 2021.

## Builds on (most cited first, 30 of 56)

- [Very Deep Convolutional Networks for Large-Scale Image Recognition](https://mlsanity.com/paper/1409.1556/) (2014)
- [A Simple Framework for Contrastive Learning of Visual Representations](https://mlsanity.com/paper/2002.05709/) (2020)
- [Explaining and Harnessing Adversarial Examples](https://mlsanity.com/paper/1412.6572/) (2014)
- [Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift](https://mlsanity.com/paper/1502.03167/) (2015)
- [Towards Deep Learning Models Resistant to Adversarial Attacks](https://mlsanity.com/paper/1706.06083/) (2017)
- [Simple and Scalable Predictive Uncertainty Estimation using Deep Ensembles](https://mlsanity.com/paper/1612.01474/) (2016)
- [Deep Learning Face Attributes in the Wild](https://mlsanity.com/paper/1411.7766/) (2014)
- [FixMatch: Simplifying Semi-Supervised Learning with Consistency and Confidence](https://mlsanity.com/paper/2001.07685/) (2020)
- [Shortcut Learning in Deep Neural Networks](https://mlsanity.com/paper/2004.07780/) (2020)
- [Invariant Risk Minimization](https://mlsanity.com/paper/1907.02893/) (2019)
- [A Simple Unified Framework for Detecting Out-of-Distribution Samples and Adversarial Attacks](https://mlsanity.com/paper/1807.03888/) (2018)
- [ImageNet-trained CNNs are biased towards texture; increasing shape bias improves accuracy and robustness](https://mlsanity.com/paper/1811.12231/) (2018)
- [MixMatch: A Holistic Approach to Semi-Supervised Learning](https://mlsanity.com/paper/1905.02249/) (2019)
- [Certified Adversarial Robustness via Randomized Smoothing](https://mlsanity.com/paper/1902.02918/) (2019)
- [Self-training with Noisy Student improves ImageNet classification](https://mlsanity.com/paper/1911.04252/) (2019)
- [Distributionally Robust Neural Networks for Group Shifts: On the Importance of Regularization for Worst-Case Generalization](https://mlsanity.com/paper/1911.08731/) (2019)
- [Deep Image Prior](https://mlsanity.com/paper/1711.10925/) (2017)
- [Co-teaching: Robust Training of Deep Neural Networks with Extremely Noisy Labels](https://mlsanity.com/paper/1804.06872/) (2018)
- [Temporal Ensembling for Semi-Supervised Learning](https://mlsanity.com/paper/1610.02242/) (2016)
- [Learning Robust Global Representations by Penalizing Local Predictive Power](https://mlsanity.com/paper/1905.13549/) (2019)
- [Adversarial Examples Are Not Bugs, They Are Features](https://mlsanity.com/paper/1905.02175/) (2019)
- [Virtual Adversarial Training: A Regularization Method for Supervised and Semi-Supervised Learning](https://mlsanity.com/paper/1704.03976/) (2017)
- [Robustness May Be at Odds with Accuracy](https://mlsanity.com/paper/1805.12152/) (2018)
- [Annotation Artifacts in Natural Language Inference Data](https://mlsanity.com/paper/1803.02324/) (2018)
- [Recognition in Terra Incognita](https://mlsanity.com/paper/1807.04975/) (2018)
- [Pseudo-Labeling and Confirmation Bias in Deep Semi-Supervised Learning](https://mlsanity.com/paper/1908.02983/) (2019)
- [Certified Robustness to Adversarial Examples with Differential Privacy](https://mlsanity.com/paper/1802.03471/) (2018)
- [Domain Generalization by Solving Jigsaw Puzzles](https://mlsanity.com/paper/1903.06864/) (2019)
- [DeCAF: A Deep Convolutional Activation Feature for Generic Visual Recognition](https://mlsanity.com/paper/1310.1531/) (2013)
- [Confidence Regularized Self-Training](https://mlsanity.com/paper/1908.09822/) (2019)

## Built on by (most cited first, 12 of 12)

- [Model Ratatouille: Recycling Diverse Models for Out-of-Distribution Generalization](https://mlsanity.com/paper/2212.10445/) (2022)
- [Spurious Feature Diversification Improves Out-of-distribution Generalization](https://mlsanity.com/paper/2309.17230/) (2023)
- [ImageNet-trained CNNs are not biased towards texture: Revisiting feature reliance through controlled suppression](https://mlsanity.com/paper/2509.20234/) (2025)
- [Meta-Exploiting Frequency Prior for Cross-Domain Few-Shot Learning](https://mlsanity.com/paper/2411.01432/) (2024)
- [Rapid Network Adaptation: Learning to Adapt Neural Networks Using Test-Time Feedback](https://mlsanity.com/paper/2309.15762/) (2023)
- [InBiaseD: Inductive Bias Distillation to Improve Generalization and Robustness through Shape-awareness](https://mlsanity.com/paper/2206.05846/) (2022)
- [Towards Understanding How Self-training Tolerates Data Backdoor Poisoning](https://mlsanity.com/paper/2301.08751/) (2023)
- [LSFSL: Leveraging Shape Information in Few-shot Learning](https://mlsanity.com/paper/2304.06672/) (2023)
- [Towards Brain Inspired Design for Addressing the Shortcomings of ANNs](https://mlsanity.com/paper/2307.00039/) (2023)
- [Can We Break Free from Strong Data Augmentations in Self-Supervised Learning?](https://mlsanity.com/paper/2404.09752/) (2024)
- [Diverse Audio Embeddings -- Bringing Features Back Outperforms CLAP!](https://mlsanity.com/paper/2309.08751/) (2023)
- [Something for (almost) nothing: Improving deep ensemble calibration using unlabeled data](https://mlsanity.com/paper/2310.02885/) (2023)

## Abstract (arXiv)

To improve model generalization, model designers often restrict the features that their models use, either implicitly or explicitly. In this work, we explore the design space of leveraging such feature priors by viewing them as distinct perspectives on the data. Specifically, we find that models trained with diverse sets of feature priors have less overlapping failure modes, and can thus be combined more effectively. Moreover, we demonstrate that jointly training such models on additional (unlabeled) data allows them to correct each other's mistakes, which, in turn, leads to better generalization and resilience to spurious correlations. Code available at https://github.com/MadryLab/copriors