# EEC: Learning to Encode and Regenerate Images for Continual Learning

arXiv: https://arxiv.org/abs/2101.04904 · HTML: https://mlsanity.com/paper/2101.04904/

## Key facts

- Cited by 40 papers in the map (data through 14 Oct 2026).
- Builds on 20 of its 32 references that are in the map.
- First posted on arXiv 13 Jan 2021.

## Builds on (most cited first, 20 of 20)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [Distilling the Knowledge in a Neural Network](https://mlsanity.com/paper/1503.02531/) (2015)
- [ImageNet Large Scale Visual Recognition Challenge](https://mlsanity.com/paper/1409.0575/) (2014)
- [Overcoming catastrophic forgetting in neural networks](https://mlsanity.com/paper/1612.00796/) (2016)
- [Fast R-CNN](https://mlsanity.com/paper/1504.08083/) (2015)
- [iCaRL: Incremental Classifier and Representation Learning](https://mlsanity.com/paper/1611.07725/) (2016)
- [Improved Training of Wasserstein GANs](https://mlsanity.com/paper/1704.00028/) (2017)
- [Unsupervised Representation Learning with Deep Convolutional Generative Adversarial Networks](https://mlsanity.com/paper/1511.06434/) (2015)
- [Continual Learning with Deep Generative Replay](https://mlsanity.com/paper/1705.08690/) (2017)
- [Riemannian Walk for Incremental Learning: Understanding Forgetting and Intransigence](https://mlsanity.com/paper/1801.10112/) (2018)
- [Large Scale Incremental Learning](https://mlsanity.com/paper/1905.13260/) (2019)
- [End-to-End Incremental Learning](https://mlsanity.com/paper/1807.09536/) (2018)
- [Learning to Remember: A Synaptic Plasticity Driven Framework for Continual Learning](https://mlsanity.com/paper/1904.03137/) (2019)
- [FearNet: Brain-Inspired Model for Incremental Learning](https://mlsanity.com/paper/1711.10563/) (2017)
- [Lifelong Machine Learning with Deep Streaming Linear Discriminant Analysis](https://mlsanity.com/paper/1909.01520/) (2019)
- [Continual Learning in Generative Adversarial Nets](https://mlsanity.com/paper/1705.08395/) (2017)
- [Incremental Classifier Learning with Generative Adversarial Networks](https://mlsanity.com/paper/1802.00853/) (2018)
- [Tell me what this is: Few-Shot Incremental Object Learning by a Robot](https://mlsanity.com/paper/2008.00819/) (2020)
- [Storing Encoded Episodes as Concepts for Continual Learning](https://mlsanity.com/paper/2007.06637/) (2020)

## Built on by (most cited first, 30 of 40)

- [A Comprehensive Survey of Continual Learning: Theory, Method and Application](https://mlsanity.com/paper/2302.00487/) (2023)
- [Always Be Dreaming: A New Approach for Data-Free Class-Incremental Learning](https://mlsanity.com/paper/2106.09701/) (2021)
- [CoIN: A Benchmark of Continual Instruction tuNing for Multimodel Large Language Model](https://mlsanity.com/paper/2403.08350/) (2024)
- [Recent Advances of Continual Learning in Computer Vision: An Overview](https://mlsanity.com/paper/2109.11369/) (2021)
- [How to Reuse and Compose Knowledge for a Lifetime of Tasks: A Survey on Continual Learning and Functional Composition](https://mlsanity.com/paper/2207.07730/) (2022)
- [Few-Shot Continual Active Learning by a Robot](https://mlsanity.com/paper/2210.04137/) (2022)
- [Open-World Continual Learning: Unifying Novelty Detection and Continual Learning](https://mlsanity.com/paper/2304.10038/) (2023)
- [Incremental Learning of Structured Memory via Closed-Loop Transcription](https://mlsanity.com/paper/2202.05411/) (2022)
- [Provable Contrastive Continual Learning](https://mlsanity.com/paper/2405.18756/) (2024)
- [F-SIOL-310: A Robotic Dataset and Benchmark for Few-Shot Incremental Object Learning](https://mlsanity.com/paper/2103.12242/) (2021)
- [Task Difficulty Aware Parameter Allocation & Regularization for Lifelong Learning](https://mlsanity.com/paper/2304.05288/) (2023)
- [Class Gradient Projection For Continual Learning](https://mlsanity.com/paper/2311.14905/) (2023)
- [ProtoDepth: Unsupervised Continual Depth Completion with Prototypes](https://mlsanity.com/paper/2503.12745/) (2025)
- [CorpusBrain++: A Continual Generative Pre-Training Framework for Knowledge-Intensive Language Tasks](https://mlsanity.com/paper/2402.16767/) (2024)
- [TEAM: Topological Evolution-aware Framework for Traffic Forecasting--Extended Version](https://mlsanity.com/paper/2410.19192/) (2024)
- [IB-DRR: Incremental Learning with Information-Back Discrete Representation Replay](https://mlsanity.com/paper/2104.10588/) (2021)
- [Continual Predictive Learning from Videos](https://mlsanity.com/paper/2204.05624/) (2022)
- [Continual Learning with Neuromorphic Computing: Foundations, Methods, and Emerging Applications](https://mlsanity.com/paper/2410.09218/) (2024)
- [Interactive Continual Learning Architecture for Long-Term Personalization of Home Service Robots](https://mlsanity.com/paper/2403.03462/) (2024)
- [UnCLe: Benchmarking Unsupervised Continual Learning for Depth Completion](https://mlsanity.com/paper/2410.18074/) (2024)
- [Neural Architecture Search of Deep Priors: Towards Continual Learning without Catastrophic Interference](https://mlsanity.com/paper/2104.06788/) (2021)
- [Generative Negative Replay for Continual Learning](https://mlsanity.com/paper/2204.05842/) (2022)
- [Continual Learning through Human-Robot Interaction: Human Perceptions of a Continual Learning Robot in Repeated Interactions](https://mlsanity.com/paper/2305.16332/) (2023)
- [Diffusion Model Meets Non-Exemplar Class-Incremental Learning and Beyond](https://mlsanity.com/paper/2408.02983/) (2024)
- [Towards Redundancy-Free Sub-networks in Continual Learning](https://mlsanity.com/paper/2312.00840/) (2023)
- [Learning Novel Objects Continually Through Curiosity](https://mlsanity.com/paper/2103.07758/) (2021)
- [Saliency-Augmented Memory Completion for Continual Learning](https://mlsanity.com/paper/2212.13242/) (2022)
- [CBCL-PR: A Cognitively Inspired Model for Class-Incremental Learning in Robotics](https://mlsanity.com/paper/2308.00199/) (2023)
- [CUCL: Codebook for Unsupervised Continual Learning](https://mlsanity.com/paper/2311.14911/) (2023)
- [EFC++: Elastic Feature Consolidation with Prototype Re-balancing for Cold Start Exemplar-free Incremental Learning](https://mlsanity.com/paper/2503.10439/) (2025)

## Abstract (arXiv)

The two main impediments to continual learning are catastrophic forgetting and memory limitations on the storage of data. To cope with these challenges, we propose a novel, cognitively-inspired approach which trains autoencoders with Neural Style Transfer to encode and store images. During training on a new task, reconstructed images from encoded episodes are replayed in order to avoid catastrophic forgetting. The loss function for the reconstructed images is weighted to reduce its effect during classifier training to cope with image degradation. When the system runs out of memory the encoded episodes are converted into centroids and covariance matrices, which are used to generate pseudo-images during classifier training, keeping classifier performance stable while using less memory. Our approach increases classification accuracy by 13-17% over state-of-the-art methods on benchmark datasets, while requiring 78% less storage space.