# Structured Sparse Memory for Recurrent Reasoning

arXiv: https://arxiv.org/abs/2609.33270 · HTML: https://mlsanity.com/paper/2609.33270/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 13 of its 20 references that are in the map.
- First posted on arXiv 27 Sep 2026.

## Builds on (most cited first, 13 of 13)

- [Overcoming catastrophic forgetting in neural networks](https://mlsanity.com/paper/1612.00796/) (2016)
- [FiLM: Visual Reasoning with a General Conditioning Layer](https://mlsanity.com/paper/1709.07871/) (2017)
- [Universal Transformers](https://mlsanity.com/paper/1807.03819/) (2018)
- [Explicit Inductive Bias for Transfer Learning with Convolutional Networks](https://mlsanity.com/paper/1802.01483/) (2018)
- [ARC-AGI-2: A New Challenge for Frontier AI Reasoning Systems](https://mlsanity.com/paper/2505.11831/) (2025)
- [Hierarchical Reasoning Model](https://mlsanity.com/paper/2506.21734/) (2025)
- [The ConceptARC Benchmark: Evaluating Understanding and Generalization in the ARC Domain](https://mlsanity.com/paper/2305.07141/) (2023)
- [Conditional Memory via Scalable Lookup: A New Axis of Sparsity for Large Language Models](https://mlsanity.com/paper/2601.07372/) (2026)
- [Hash Embeddings for Efficient Word Representations](https://mlsanity.com/paper/1709.03933/) (2017)
- [Addressing the Abstraction and Reasoning Corpus via Procedural Example Generation](https://mlsanity.com/paper/2404.07353/) (2024)
- [ARC Is a Vision Problem!](https://mlsanity.com/paper/2511.14761/) (2025)
- [Product of Experts with LLMs: Boosting Performance on ARC Is a Matter of Perspective](https://mlsanity.com/paper/2505.07859/) (2025)
- [Universal Reasoning Model](https://mlsanity.com/paper/2512.14693/) (2025)

## Abstract (arXiv)

Recurrent models trained from scratch have recently become competitive on ARC-style reasoning tasks, but the usual framing around small recurrent backbones overlooks two important parts of the system: task-conditioned memory and synthetic augmentation data. We study this regime through CHARM, a compact hybrid ARC model that combines recurrent reasoning with structured task memory, synthetic data, and inference-time aggregation. In existing approaches, task-conditioned memory supplies a large hidden source of capacity, reaching more than 30x the size of the recurrent backbone. We introduce a compositional sparse embedding (CoSE) for task conditioning that reduces learned task-memory parameters by over 90% while improving pass@2 in controlled ARC ablations. For the recurrent backbone, recurrent depth helps only when balanced with learning horizon. Combining these ingredients, our system reaches 84% pass@2 on ARC-AGI-1 and 46.7% pass@2 on ARC-AGI-2 public evaluation. The benefits of structured memory also generalize to unseen puzzles and other domains. Our code, dataset, and model checkpoints are available at https://github.com/water-vapor/charm.