# Unsupervised Progressive Learning and the STAM Architecture

arXiv: https://arxiv.org/abs/1904.02021 · HTML: https://mlsanity.com/paper/1904.02021/

## Key facts

- Cited by 30 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 3 Apr 2019.

## Built on by (most cited first, 30 of 30)

- [Self-Supervised Models are Continual Learners](https://mlsanity.com/paper/2112.04215/) (2021)
- [Always Be Dreaming: A New Approach for Data-Free Class-Incremental Learning](https://mlsanity.com/paper/2106.09701/) (2021)
- [Representational Continuity for Unsupervised Continual Learning](https://mlsanity.com/paper/2110.06976/) (2021)
- [A Data-Free Approach to Mitigate Catastrophic Forgetting in Federated Class Incremental Learning for Vision Tasks](https://mlsanity.com/paper/2311.07784/) (2023)
- [Memory-Efficient Semi-Supervised Continual Learning: The World is its Own Replay Buffer](https://mlsanity.com/paper/2101.09536/) (2021)
- [Unsupervised Continual Learning Via Pseudo Labels](https://mlsanity.com/paper/2104.07164/) (2021)
- [A soft nearest-neighbor framework for continual semi-supervised learning](https://mlsanity.com/paper/2212.05102/) (2022)
- [SCALE: Online Self-Supervised Lifelong Learning without Prior Knowledge](https://mlsanity.com/paper/2208.11266/) (2022)
- [Label-Efficient Learning in Agriculture: A Comprehensive Review](https://mlsanity.com/paper/2305.14691/) (2023)
- [BayesPCN: A Continually Learnable Predictive Coding Associative Memory](https://mlsanity.com/paper/2205.09930/) (2022)
- [A Domain-Agnostic Approach for Characterization of Lifelong Learning Systems](https://mlsanity.com/paper/2301.07799/) (2023)
- [SPeCiaL: Self-Supervised Pretraining for Continual Learning](https://mlsanity.com/paper/2106.09065/) (2021)
- [Model-Free Generative Replay for Lifelong Reinforcement Learning: Application to Starcraft-2](https://mlsanity.com/paper/2208.05056/) (2022)
- [Out-Of-Distribution Detection In Unsupervised Continual Learning](https://mlsanity.com/paper/2204.05462/) (2022)
- [Integrating Present and Past in Unsupervised Continual Learning](https://mlsanity.com/paper/2404.19132/) (2024)
- [Memory Storyboard: Leveraging Temporal Segmentation for Streaming Self-Supervised Learning from Egocentric Videos](https://mlsanity.com/paper/2501.12254/) (2025)
- [Lifelong Intelligence Beyond the Edge using Hyperdimensional Computing](https://mlsanity.com/paper/2403.04759/) (2024)
- [Contrastive Learning for Online Semi-Supervised General Continual Learning](https://mlsanity.com/paper/2207.05615/) (2022)
- [Unsupervised Class-Incremental Learning Through Confusion](https://mlsanity.com/paper/2104.04450/) (2021)
- [Autonomous Cross Domain Adaptation under Extreme Label Scarcity](https://mlsanity.com/paper/2209.01548/) (2022)
- [U-TELL: Unsupervised Task Expert Lifelong Learning](https://mlsanity.com/paper/2405.14623/) (2024)
- [PhiNets: Brain-inspired Non-contrastive Learning Based on Temporal Prediction Hypothesis](https://mlsanity.com/paper/2405.14650/) (2024)
- [Domain-Aware Augmentations for Unsupervised Online General Continual Learning](https://mlsanity.com/paper/2309.06896/) (2023)
- [Forward-Backward Knowledge Distillation for Continual Clustering](https://mlsanity.com/paper/2405.19234/) (2024)
- [Unsupervised Few-Shot Continual Learning for Remote Sensing Image Scene Classification](https://mlsanity.com/paper/2406.18574/) (2024)
- [Continual Learning by Three-Phase Consolidation](https://mlsanity.com/paper/2403.14679/) (2024)
- [Patch-Based Contrastive Learning and Memory Consolidation for Online Unsupervised Continual Learning](https://mlsanity.com/paper/2409.16391/) (2024)
- [Caption, Create, Continue: Continual Learning with Pre-trained Generative Vision-Language Models](https://mlsanity.com/paper/2409.17806/) (2024)
- [Unsupervised Continual Clustering via Forward-Backward Knowledge Distillation](https://mlsanity.com/paper/2606.07474/) (2026)
- [CLIMB: Centroid-Based Hierarchical Memory for Online Continual Self-Supervised Learning](https://mlsanity.com/paper/2606.31275/) (2026)

## Abstract (arXiv)

We first pose the Unsupervised Progressive Learning (UPL) problem: an online representation learning problem in which the learner observes a non-stationary and unlabeled data stream, learning a growing number of features that persist over time even though the data is not stored or replayed. To solve the UPL problem we propose the Self-Taught Associative Memory (STAM) architecture. Layered hierarchies of STAM modules learn based on a combination of online clustering, novelty detection, forgetting outliers, and storing only prototypical features rather than specific examples. We evaluate STAM representations using clustering and classification tasks. While there are no existing learning scenarios that are directly comparable to UPL, we compare the STAM architecture with two recent continual learning models, Memory Aware Synapses (MAS) and Gradient Episodic Memories (GEM), after adapting them in the UPL setting.