mlsanityData to 14 Oct 2026

The no-free-lunch theorems of supervised learning

arXiv 2202.04513 ↗First posted 9 Feb 2022cs.LGcited by 16

Not on the map. Why?

Abstract (arXiv)

The no-free-lunch theorems promote a skeptical conclusion that all possible machine learning algorithms equally lack justification. But how could this leave room for a learning theory, that shows that some algorithms are better than others? Drawing parallels to the philosophy of induction, we point out that the no-free-lunch results presuppose a conception of learning algorithms as purely data-driven. On this conception, every algorithm must have an inherent inductive bias, that wants justification. We argue that many standard learning algorithms should rather be understood as model-dependent: in each application they also require for input a model, representing a bias. Generic algorithms themselves, they can be given a model-relative justification.

Key facts. Cited by 16 papers in the map (data through 14 Oct 2026). Builds on 13 of its 108 references that are in the map. First posted on arXiv 9 Feb 2022.

Builds on

13 of 108 references.

References · 13

  1. 2016Understanding deep learning requires rethinking generalizationcited by 1,674
  2. 2017A Closer Look at Memorization in Deep Networkscited by 942
  3. 2017Exploring Generalization in Deep Learningcited by 600
  4. 2017Computing Nonvacuous Generalization Bounds for Deep (Stochastic) Neural Networks with Many More Parameters than Training Datacited by 533
  5. 2015Norm-Based Capacity Control in Neural Networkscited by 299
  6. 2017Generalization in Deep Learningcited by 146
  7. 2014Inconsistency of Bayesian Inference for Misspecified Linear Models, and a Proposal for Repairing Itcited by 95
  8. 2019On the marginal likelihood and cross-validationcited by 37
  9. 2019Minimum Description Length Revisitedcited by 37
  10. 2016Fast Rates for General Unbounded Loss Functions: from ERM to Generalized Bayescited by 27
  11. 2019Minimizers of the Empirical Risk and Risk Monotonicitycited by 11
  12. 2020What is important about the No Free Lunch theorems?cited by 9
1 more
  1. 2020Absolutely No Free Lunches!cited by 4

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2022: 2; 2023: 5; 2024: 1; 2025: 3; 2026: 5.

16 papers.

Sort:

  1. 2022Scientific Inference With Interpretable Machine Learning: Analyzing Models to Learn About Real-World Phenomenacited by 21
  2. 2025The Benchmarking Epistemology: Validity Theory for Evaluating Machine Learning Modelscited by 13
  3. 2022ML Interpretability: Simple Isn't Easycited by 7
  4. 2023Statistical learning theory and Occam's razor: The core argumentcited by 5
  5. 2023Relax, it doesn't matter how you get there: A new self-supervised approach for multi-timescale behavior analysiscited by 4
  6. 2024A General Theory for Compositional Generalizationcited by 4
  7. 2023On the Computational Complexity of Ethics: Moral Tractability for Minds and Machinescited by 2
  8. 2026Solomonoff inductioncited by 2
  9. 2023Transgressing the boundaries: towards a rigorous understanding of deep learning and its (non-)robustnesscited by 1
  10. 2025Generating Skyline Datasets for Data Science Modelscited by 1
  11. 2023When Is Inductive Inference Possible?cited by 0
  12. 2025The Knowledge-Behaviour Disconnect in LLM-based Chatbotscited by 0
4 more
  1. 2026How Complexity Contributes to Learning Opacity in Machine Learningcited by 0
  2. 2026Benign interpolation and Occam's razorcited by 0
  3. 2026Statistical learning theory and Occam's razor: Regularizationcited by 0
  4. 2026Understanding Generalization Requires Universal Inductioncited by 0