# Distinguishing rule- and exemplar-based generalization in learning systems

arXiv: https://arxiv.org/abs/2110.04328 · HTML: https://mlsanity.com/paper/2110.04328/

## Key facts

- Cited by 12 papers in the map (data through 14 Oct 2026).
- Builds on 36 of its 61 references that are in the map.
- First posted on arXiv 8 Oct 2021.

## Builds on (most cited first, 30 of 36)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [A Simple Framework for Contrastive Learning of Visual Representations](https://mlsanity.com/paper/2002.05709/) (2020)
- [Deep Learning Face Attributes in the Wild](https://mlsanity.com/paper/1411.7766/) (2014)
- [Neural Tangent Kernel: Convergence and Generalization in Neural Networks](https://mlsanity.com/paper/1806.07572/) (2018)
- [Visualizing and Understanding Convolutional Networks](https://mlsanity.com/paper/1311.2901/) (2013)
- [Shortcut Learning in Deep Neural Networks](https://mlsanity.com/paper/2004.07780/) (2020)
- [Invariant Risk Minimization](https://mlsanity.com/paper/1907.02893/) (2019)
- [Understanding deep learning requires rethinking generalization](https://mlsanity.com/paper/1611.03530/) (2016)
- [CLEVR: A Diagnostic Dataset for Compositional Language and Elementary Visual Reasoning](https://mlsanity.com/paper/1612.06890/) (2016)
- [ImageNet-trained CNNs are biased towards texture; increasing shape bias improves accuracy and robustness](https://mlsanity.com/paper/1811.12231/) (2018)
- [Model Cards for Model Reporting](https://mlsanity.com/paper/1810.03993/) (2018)
- [A Closer Look at Memorization in Deep Networks](https://mlsanity.com/paper/1706.05394/) (2017)
- [A Convergence Theory for Deep Learning via Over-Parameterization](https://mlsanity.com/paper/1811.03962/) (2018)
- [Gradient Descent Provably Optimizes Over-parameterized Neural Networks](https://mlsanity.com/paper/1810.02054/) (2018)
- [Recognition in Terra Incognita](https://mlsanity.com/paper/1807.04975/) (2018)
- [Zero-Shot Learning -- A Comprehensive Evaluation of the Good, the Bad and the Ugly](https://mlsanity.com/paper/1707.00600/) (2017)
- [What Neural Networks Memorize and Why: Discovering the Long Tail via Influence Estimation](https://mlsanity.com/paper/2008.03703/) (2020)
- [Underspecification Presents Challenges for Credibility in Modern Machine Learning](https://mlsanity.com/paper/2011.03395/) (2020)
- [Does Learning Require Memorization? A Short Tale about a Long Tail](https://mlsanity.com/paper/1906.05271/) (2019)
- [Neural Module Networks](https://mlsanity.com/paper/1511.02799/) (2015)
- [The Effectiveness of Data Augmentation in Image Classification using Deep Learning](https://mlsanity.com/paper/1712.04621/) (2017)
- [An Investigation of Why Overparameterization Exacerbates Spurious Correlations](https://mlsanity.com/paper/2005.04345/) (2020)
- [On the Expressive Power of Deep Neural Networks](https://mlsanity.com/paper/1606.05336/) (2016)
- [Data Augmentation Generative Adversarial Networks](https://mlsanity.com/paper/1711.04340/) (2017)
- [Learning Overparameterized Neural Networks via Stochastic Gradient Descent on Structured Data](https://mlsanity.com/paper/1808.01204/) (2018)
- [Deep Learning: A Critical Appraisal](https://mlsanity.com/paper/1801.00631/) (2018)
- [Measuring abstract reasoning in neural networks](https://mlsanity.com/paper/1807.04225/) (2018)
- [Visualizing and Understanding Recurrent Networks](https://mlsanity.com/paper/1506.02078/) (2015)
- [A Meta-Transfer Objective for Learning to Disentangle Causal Mechanisms](https://mlsanity.com/paper/1901.10912/) (2019)
- [Understanding and Mitigating the Tradeoff Between Robustness and Accuracy](https://mlsanity.com/paper/2002.10716/) (2020)

## Built on by (most cited first, 12 of 12)

- [How much do language models copy from their training data? Evaluating linguistic novelty in text generation using RAVEN](https://mlsanity.com/paper/2111.09509/) (2021)
- [In-Context Learning Learns Label Relationships but Is Not Conventional Learning](https://mlsanity.com/paper/2307.12375/) (2023)
- [Mechanistic Mode Connectivity](https://mlsanity.com/paper/2211.08422/) (2022)
- [Transformers generalize differently from information stored in context vs in weights](https://mlsanity.com/paper/2210.05675/) (2022)
- [Measuring Inductive Biases of In-Context Learning with Underspecified Demonstrations](https://mlsanity.com/paper/2305.13299/) (2023)
- [Closing the Gap between TD Learning and Supervised Learning -- A Generalisation Point of View](https://mlsanity.com/paper/2401.11237/) (2024)
- [When does compositional structure yield compositional generalization? A kernel theory](https://mlsanity.com/paper/2405.16391/) (2024)
- [Toward Operationalizing Pipeline-aware ML Fairness: A Research Agenda for Developing Practical Guidelines and Tools](https://mlsanity.com/paper/2309.17337/) (2023)
- [A Theory of Initialisation's Impact on Specialisation](https://mlsanity.com/paper/2503.02526/) (2025)
- [Compositional Generalisation for Explainable Hate Speech Detection](https://mlsanity.com/paper/2506.03916/) (2025)
- [Investigating Concept Alignment Using Implausible Category Members](https://mlsanity.com/paper/2605.21683/) (2026)
- [LLMs Learn Better In-Context from Rules than from Examples](https://mlsanity.com/paper/2609.03213/) (2026)

## Abstract (arXiv)

Machine learning systems often do not share the same inductive biases as humans and, as a result, extrapolate or generalize in ways that are inconsistent with our expectations. The trade-off between exemplar- and rule-based generalization has been studied extensively in cognitive psychology; in this work, we present a protocol inspired by these experimental approaches to probe the inductive biases that control this tradeoff in category-learning systems. We isolate two such inductive biases: feature-level bias (differences in which features are more readily learned) and exemplar or rule bias (differences in how these learned features are used for generalization). We find that standard neural network models are feature-biased and exemplar-based, and discuss the implications of these findings for machine learning research on systematic generalization, fairness, and data augmentation.