Hard ASH: Sparsity and the right optimizer make a continual learner
Not on the map. Why?
Abstract (arXiv)
In class incremental learning, neural networks typically suffer from catastrophic forgetting. We show that an MLP featuring a sparse activation function and an adaptive learning rate optimizer can compete with established regularization techniques in the Split-MNIST task. We highlight the effectiveness of the Adaptive SwisH (ASH) activation function in this context and introduce a novel variant, Hard Adaptive SwisH (Hard ASH) to further enhance the learning retention.
Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Builds on 13 of its 20 references that are in the map. First posted on arXiv 26 Apr 2024.
Builds on
13 of 20 references.
References · 13
- 2016Overcoming catastrophic forgetting in neural networkscited by 4,718
- 2015Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classificationcited by 3,590
- 2017Memory Aware Synapses: Learning what (not) to forgetcited by 1,094
- 2015BinaryConnect: Training Deep Neural Networks with binary weights during propagationscited by 571
- 2016Weight Normalization: A Simple Reparameterization to Accelerate Training of Deep Neural Networkscited by 541
- 2018Progress & Compress: A scalable framework for continual learningcited by 516
- 2018Re-evaluating Continual Learning Scenarios: A Categorization and Case for Strong Baselinescited by 228
- 2013k-Sparse Autoencoderscited by 209
- 2020Descending through a Crowded Valley - Benchmarking Deep Learning Optimizerscited by 118
- 2019How Can We Be So Dense? The Benefits of Using Highly Sparse Representationscited by 45
- 2020A Practical Sparse Approximation for Real Time Recurrent Learningcited by 18
- 2021Algorithmic insights on continual learning from fruit fliescited by 11
1 more
- 2023Elephant Neural Networks: Born to Be a Continual Learnercited by 10
Built on by
Citing papers per year: 2024: 1.
1 paper.