# Efficient Induction of Language Models Via Probabilistic Concept Formation

arXiv: https://arxiv.org/abs/2212.11937 · HTML: https://mlsanity.com/paper/2212.11937/

## Key facts

- Cited by 9 papers in the map (data through 14 Oct 2026).
- Builds on 6 of its 20 references that are in the map.
- First posted on arXiv 22 Dec 2022.

## Builds on (most cited first, 6 of 6)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Efficient Estimation of Word Representations in Vector Space](https://mlsanity.com/paper/1301.3781/) (2013)
- [Distributed Representations of Words and Phrases and their Compositionality](https://mlsanity.com/paper/1310.4546/) (2013)
- [Downstream Datasets Make Surprisingly Good Pretraining Corpora](https://mlsanity.com/paper/2209.14389/) (2022)
- [Convolutional Cobweb: A Model of Incremental Learning from 2D Images](https://mlsanity.com/paper/2201.06740/) (2022)

## Built on by (most cited first, 9 of 9)

- [Incremental Concept Formation over Visual Images Without Catastrophic Forgetting](https://mlsanity.com/paper/2402.16933/) (2024)
- [Cobweb: An Incremental and Hierarchical Model of Human-Like Category Learning](https://mlsanity.com/paper/2403.03835/) (2024)
- [Taxonomic Networks: A Representation for Neuro-Symbolic Pairing](https://mlsanity.com/paper/2505.24601/) (2025)
- [Hierarchical Semantic Retrieval with Cobweb](https://mlsanity.com/paper/2510.02539/) (2025)
- [Incremental and Data-Efficient Concept Formation to Support Masked Word Prediction](https://mlsanity.com/paper/2409.12440/) (2024)
- [Explaining Robustness to Catastrophic Forgetting Through Incremental Concept Formation](https://mlsanity.com/paper/2510.23756/) (2025)
- [CobwebTM: Probabilistic Concept Formation for Lifelong and Hierarchical Topic Modeling](https://mlsanity.com/paper/2604.14489/) (2026)
- [Diffusion Models and Concept Formation](https://mlsanity.com/paper/2609.13047/) (2026)
- [A Unified Account of Concepts and Chunks](https://mlsanity.com/paper/2609.30414/) (2026)

## Abstract (arXiv)

This paper presents a novel approach to the acquisition of language models from corpora. The framework builds on Cobweb, an early system for constructing taxonomic hierarchies of probabilistic concepts that used a tabular, attribute-value encoding of training cases and concepts, making it unsuitable for sequential input like language. In response, we explore three new extensions to Cobweb -- the Word, Leaf, and Path variants. These systems encode each training case as an anchor word and surrounding context words, and they store probabilistic descriptions of concepts as distributions over anchor and context information. As in the original Cobweb, a performance element sorts a new instance downward through the hierarchy and uses the final node to predict missing features. Learning is interleaved with performance, updating concept probabilities and hierarchy structure as classification occurs. Thus, the new approaches process training cases in an incremental, online manner that it very different from most methods for statistical language learning. We examine how well the three variants place synonyms together and keep homonyms apart, their ability to recall synonyms as a function of training set size, and their training efficiency. Finally, we discuss related work on incremental learning and directions for further research.