mlsanityData to 14 Oct 2026

ATLAS: Universal Function Approximator for Memory Retention

arXiv 2208.05388 ↗First posted 10 Aug 2022cs.LG cs.NEcited by 0

Not on the map. Why?

Abstract (arXiv)

Artificial neural networks (ANNs), despite their universal function approximation capability and practical success, are subject to catastrophic forgetting. Catastrophic forgetting refers to the abrupt unlearning of a previous task when a new task is learned. It is an emergent phenomenon that hinders continual learning. Existing universal function approximation theorems for ANNs guarantee function approximation ability, but do not predict catastrophic forgetting. This paper presents a novel universal approximation theorem for multi-variable functions using only single-variable functions and exponential functions. Furthermore, we present ATLAS: a novel ANN architecture based on the new theorem. It is shown that ATLAS is a universal function approximator capable of some memory retention, and continual learning. The memory of ATLAS is imperfect, with some off-target effects during continual learning, but it is well-behaved and predictable. An efficient implementation of ATLAS is provided. Experiments are conducted to evaluate both the function approximation and memory retention capabilities of ATLAS.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 14 of its 29 references that are in the map. First posted on arXiv 10 Aug 2022.

Builds on

14 of 29 references.

References · 14

  1. 2016Overcoming catastrophic forgetting in neural networkscited by 4,787
  2. 2016Progressive Neural Networkscited by 1,361
  3. 2019Orthogonal Gradient Descent for Continual Learningcited by 369
  4. 2017Measuring Catastrophic Forgetting in Neural Networkscited by 360
  5. 2017Universal Function Approximation by Deep Neural Nets with Bounded Width and ReLU Activationscited by 104
  6. 2020A Theoretical Analysis of Catastrophic Forgetting through the NTK Overlap Matrixcited by 103
  7. 2019Continual Learning via Neural Pruningcited by 101
  8. 2020Generalisation Guarantees for Continual Learning with Orthogonal Gradient Descentcited by 68
  9. 2020A Theoretical Framework for Target Propagationcited by 57
  10. 2021Understanding Catastrophic Forgetting and Remembering in Continual Learning with Optimal Relevance Mappingcited by 48
  11. 2020Extreme Memorization via Scale of Initializationcited by 18
  12. 2021Functional Regularization for Reinforcement Learning via Learned Fourier Featurescited by 17
2 more
  1. 2016Learning activation functions from data using cubic spline interpolationcited by 13
  2. 2020Meta Continual Learning via Dynamic Programmingcited by 4

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2022KASAM: Spline Additive Models for Function Approximation
  2. 2021Statistical Mechanical Analysis of Catastrophic Forgetting in Continual Learning with Teacher and Student Networks
  3. 2022Maslow's Hammer for Catastrophic Forgetting: Node Re-Use vs Node Activation
  4. 2021Continual Learning via Bit-Level Information Preserving
  5. 2021In Defense of the Learning Without Forgetting for Task Incremental Learning
  6. 2024Optimal Protocols for Continual Learning via Statistical Physics and Control Theory
  7. 2026Putting a Face to Forgetting: Continual Learning meets Mechanistic Interpretability
  8. 2026To Retain or to Adapt? Generalizing Continual Learning