# Statistical learning does not always entail knowledge

arXiv: https://arxiv.org/abs/2501.01963 · HTML: https://mlsanity.com/paper/2501.01963/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 16 of its 66 references that are in the map.
- First posted on arXiv 17 Dec 2024.

## Builds on (most cited first, 16 of 16)

- [On the Opportunities and Risks of Foundation Models](https://mlsanity.com/paper/2108.07258/) (2021)
- [Convex optimization](https://mlsanity.com/paper/2106.01946/) (2021)
- [Retrieval-Augmented Generation for Large Language Models: A Survey](https://mlsanity.com/paper/2312.10997/) (2023)
- [A Survey on Hallucination in Large Language Models: Principles, Taxonomy, Challenges, and Open Questions](https://mlsanity.com/paper/2311.05232/) (2023)
- [Proving Test Set Contamination in Black Box Language Models](https://mlsanity.com/paper/2310.17623/) (2023)
- [Statistical physics of inference: Thresholds and algorithms](https://mlsanity.com/paper/1511.02476/) (2015)
- [Synthetic Data Generation with Large Language Models for Text Classification: Potential and Limitations](https://mlsanity.com/paper/2310.07849/) (2023)
- [Large language model validity via enhanced conformal prediction methods](https://mlsanity.com/paper/2406.09714/) (2024)
- [The Futility of Bias-Free Learning and Search](https://mlsanity.com/paper/1907.06010/) (2019)
- [Bits from Biology for Computational Intelligence](https://mlsanity.com/paper/1412.0291/) (2014)
- [High-dimensional inference: a statistical mechanics perspective](https://mlsanity.com/paper/2010.14863/) (2020)
- [Mode hunting through active information](https://mlsanity.com/paper/2011.05794/) (2020)
- [Is it possible to know cosmological fine-tuning?](https://mlsanity.com/paper/2401.04190/) (2024)
- [The Famine of Forte: Few Search Problems Greatly Favor Your Algorithm](https://mlsanity.com/paper/1609.08913/) (2016)
- [A simple correction for COVID-19 sampling bias](https://mlsanity.com/paper/2007.07426/) (2020)
- [High Dimensional Mode Hunting Using Pettiest Components Analysis](https://mlsanity.com/paper/2101.04288/) (2021)

## Built on by (most cited first, 1 of 1)

- [PRIM-cipal components analysis](https://mlsanity.com/paper/2604.15538/) (2026)

## Abstract (arXiv)

In this paper, we study learning and knowledge acquisition (LKA) of an agent about a proposition that is either true or false. We use a Bayesian approach, where the agent receives data to update his beliefs about the proposition according to a posterior distribution. The LKA is formulated in terms of active information, with data representing external or exogenous information that modifies the agent's beliefs. It is assumed that data provide details about a number of features that are relevant to the proposition. We show that this leads to a Gibbs distribution posterior, which is in maximum entropy relative to the prior, conditioned on the side constraints that the data provide in terms of the features. We demonstrate that full learning is sometimes not possible and full knowledge acquisition is never possible when the number of extracted features is too small. We also distinguish between primary learning (receiving data about features of relevance for the proposition) and secondary learning (receiving data about the learning of another agent). We argue that this type of secondary learning does not represent true knowledge acquisition. Our results have implications for statistical learning algorithms, and we claim that such algorithms do not always generate true knowledge. The theory is illustrated with several examples.