# Building, Reusing, and Generalizing Abstract Representations from Concrete Sequences

arXiv: https://arxiv.org/abs/2410.21332 · HTML: https://mlsanity.com/paper/2410.21332/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 8 of its 74 references that are in the map.
- First posted on arXiv 27 Oct 2024.

## Builds on (most cited first, 8 of 8)

- [Grokking: Generalization Beyond Overfitting on Small Algorithmic Datasets](https://mlsanity.com/paper/2201.02177/) (2022)
- [Using cognitive psychology to understand GPT-3](https://mlsanity.com/paper/2206.14576/) (2022)
- [On Interpretability of Artificial Neural Networks: A Survey](https://mlsanity.com/paper/2001.02522/) (2020)
- [A standardized Project Gutenberg corpus for statistical analysis of natural language and quantitative linguistics](https://mlsanity.com/paper/1812.08092/) (2018)
- [Call for Papers -- The BabyLM Challenge: Sample-efficient pretraining on a developmentally plausible corpus](https://mlsanity.com/paper/2301.11796/) (2023)
- [Probabilistic Analogical Mapping with Semantic Relation Networks](https://mlsanity.com/paper/2103.16704/) (2021)
- [Evaluating Understanding on Conceptual Abstraction Benchmarks](https://mlsanity.com/paper/2206.14187/) (2022)
- [Abstract representations of events arise from mental errors in learning and memory](https://mlsanity.com/paper/1805.12491/) (2018)

## Built on by (most cited first, 2 of 2)

- [From Tokens to Thoughts: How LLMs and Humans Trade Compression for Meaning](https://mlsanity.com/paper/2505.17117/) (2025)
- [Incremental Learning of Sparse Attention Patterns in Transformers](https://mlsanity.com/paper/2602.19143/) (2026)

## Abstract (arXiv)

Humans excel at learning abstract patterns across different sequences, filtering out irrelevant details, and transferring these generalized concepts to new sequences. In contrast, many sequence learning models lack the ability to abstract, which leads to memory inefficiency and poor transfer. We introduce a non-parametric hierarchical variable learning model (HVM) that learns chunks from sequences and abstracts contextually similar chunks as variables. HVM efficiently organizes memory while uncovering abstractions, leading to compact sequence representations. When learning on language datasets such as babyLM, HVM learns a more efficient dictionary than standard compression algorithms such as Lempel-Ziv. In a sequence recall task requiring the acquisition and transfer of variables embedded in sequences, we demonstrate HVM's sequence likelihood correlates with human recall times. In contrast, large language models (LLMs) struggle to transfer abstract variables as effectively as humans. From HVM's adjustable layer of abstraction, we demonstrate that the model realizes a precise trade-off between compression and generalization. Our work offers a cognitive model that captures the learning and transfer of abstract representations in human cognition and differentiates itself from LLMs.