mlsanityData to 7 Oct 2026

Distinguishing rule- and exemplar-based generalization in learning systems

arXiv 2110.04328 ↗First posted 8 Oct 2021cs.LGcited by 12

Not on the map. Why?

Abstract (arXiv)

Machine learning systems often do not share the same inductive biases as humans and, as a result, extrapolate or generalize in ways that are inconsistent with our expectations. The trade-off between exemplar- and rule-based generalization has been studied extensively in cognitive psychology; in this work, we present a protocol inspired by these experimental approaches to probe the inductive biases that control this tradeoff in category-learning systems. We isolate two such inductive biases: feature-level bias (differences in which features are more readily learned) and exemplar or rule bias (differences in how these learned features are used for generalization). We find that standard neural network models are feature-biased and exemplar-based, and discuss the implications of these findings for machine learning research on systematic generalization, fairness, and data augmentation.

Key facts. Cited by 12 papers in the map (data through 7 Oct 2026). Builds on 36 of its 61 references that are in the map. First posted on arXiv 8 Oct 2021.

Builds on

36 of 61 references.

References · 36

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 11,979
  3. 2014Deep Learning Face Attributes in the Wildcited by 3,679
  4. 2018Neural Tangent Kernel: Convergence and Generalization in Neural Networkscited by 2,597
  5. 2013Visualizing and Understanding Convolutional Networkscited by 2,278
  6. 2020Shortcut Learning in Deep Neural Networkscited by 1,928
  7. 2019Invariant Risk Minimizationcited by 1,832
  8. 2016Understanding deep learning requires rethinking generalizationcited by 1,657
  9. 2016CLEVR: A Diagnostic Dataset for Compositional Language and Elementary Visual Reasoningcited by 1,445
  10. 2018ImageNet-trained CNNs are biased towards texture; increasing shape bias improves accuracy and robustnesscited by 1,430
  11. 2018Model Cards for Model Reportingcited by 1,068
  12. 2017A Closer Look at Memorization in Deep Networkscited by 936
24 more
  1. 2018A Convergence Theory for Deep Learning via Over-Parameterizationcited by 784
  2. 2018Recognition in Terra Incognitacited by 655
  3. 2018Gradient Descent Provably Optimizes Over-parameterized Neural Networkscited by 655
  4. 2017Zero-Shot Learning -- A Comprehensive Evaluation of the Good, the Bad and the Uglycited by 536
  5. 2020What Neural Networks Memorize and Why: Discovering the Long Tail via Influence Estimationcited by 492
  6. 2020Underspecification Presents Challenges for Credibility in Modern Machine Learningcited by 488
  7. 2019Does Learning Require Memorization? A Short Tale about a Long Tailcited by 452
  8. 2015Neural Module Networkscited by 375
  9. 2017The Effectiveness of Data Augmentation in Image Classification using Deep Learningcited by 368
  10. 2020An Investigation of Why Overparameterization Exacerbates Spurious Correlationscited by 310
  11. 2016On the Expressive Power of Deep Neural Networkscited by 302
  12. 2017Data Augmentation Generative Adversarial Networkscited by 278
  13. 2018Learning Overparameterized Neural Networks via Stochastic Gradient Descent on Structured Datacited by 260
  14. 2018Deep Learning: A Critical Appraisalcited by 214
  15. 2018Measuring abstract reasoning in neural networkscited by 165
  16. 2015Visualizing and Understanding Recurrent Networkscited by 158
  17. 2019A Meta-Transfer Objective for Learning to Disentangle Causal Mechanismscited by 158
  18. 2020Understanding and Mitigating the Tradeoff Between Robustness and Accuracycited by 148
  19. 2020When is Memorization of Irrelevant Training Data Necessary for High-Accuracy Learning?cited by 102
  20. 2019Measuring Compositionality in Representation Learningcited by 84
  21. 2019Environmental drivers of systematicity and generalization in a situated agentcited by 50
  22. 2017Cognitive Psychology for Deep Neural Networks: A Shape Bias Case Studycited by 46
  23. 2017Generalization without systematicity: On the compositional skills of sequence-to-sequence recurrent networkscited by 18
  24. 2019Analyzing machine-learned representations: A natural language case studycited by 5

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 1; 2022: 2; 2023: 3; 2024: 2; 2025: 2; 2026: 2.

12 papers.

  1. 2021How much do language models copy from their training data? Evaluating linguistic novelty in text generation using RAVENcited by 124
  2. 2022Mechanistic Mode Connectivitycited by 50
  3. 2022Transformers generalize differently from information stored in context vs in weightscited by 49
  4. 2023Measuring Inductive Biases of In-Context Learning with Underspecified Demonstrationscited by 49
  5. 2023In-Context Learning Learns Label Relationships but Is Not Conventional Learningcited by 48
  6. 2024Closing the Gap between TD Learning and Supervised Learning -- A Generalisation Point of Viewcited by 26
  7. 2024When does compositional structure yield compositional generalization? A kernel theorycited by 19
  8. 2023Toward Operationalizing Pipeline-aware ML Fairness: A Research Agenda for Developing Practical Guidelines and Toolscited by 7
  9. 2025A Theory of Initialisation's Impact on Specialisationcited by 6
  10. 2025Compositional Generalisation for Explainable Hate Speech Detectioncited by 1
  11. 2026Investigating Concept Alignment Using Implausible Category Memberscited by 0
  12. 2026LLMs Learn Better In-Context from Rules than from Examplescited by 0