mlsanityData to 14 Oct 2026

Continual Learning of Context-dependent Processing in Neural Networks

arXiv 1810.01256 ↗First posted 29 Sep 2018cs.LG cs.AI cs.CVcited by 179

Not on the map. Why?

Abstract (arXiv)

Deep neural networks (DNNs) are powerful tools in learning sophisticated but fixed mapping rules between inputs and outputs, thereby limiting their application in more complex and dynamic situations in which the mapping rules are not kept the same but changing according to different contexts. To lift such limits, we developed a novel approach involving a learning algorithm, called orthogonal weights modification (OWM), with the addition of a context-dependent processing (CDP) module. We demonstrated that with OWM to overcome the problem of catastrophic forgetting, and the CDP module to learn how to reuse a feature representation and a classifier for different contexts, a single network can acquire numerous context-dependent mapping rules in an online and continual manner, with as few as ~10 samples to learn each. This should enable highly compact systems to gradually learn myriad regularities of the real world and eventually behave appropriately within it.

Key facts. Cited by 179 papers in the map (data through 14 Oct 2026). First posted on arXiv 29 Sep 2018.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2021: 14; 2022: 25; 2023: 39; 2024: 35; 2025: 39; 2026: 27.

179 papers.

Sort:

By area:

  1. 2023A Comprehensive Survey of Continual Learning: Theory, Method and Applicationcited by 872
  2. 2022Efficient Test-Time Model Adaptation without Forgettingcited by 469
  3. 2022S-Prompts Learning with Pre-trained Transformers: An Occam's Razor for Domain Incremental Learningcited by 246
  4. 2024AlphaEdit: Null-Space Constrained Knowledge Editing for Language Modelscited by 236
  5. 2023Class-Incremental Learning: A Surveycited by 209
  6. 2021Training Networks in Null Space of Feature Covariance for Continual Learningcited by 143
  7. 2022Continual Learning of Natural Language Processing Tasks: A Surveycited by 83
  8. 2022TRGP: Trust Region Gradient Projection for Continual Learningcited by 82
  9. 2024Continual Learning and Catastrophic Forgettingcited by 77
  10. 2023Multimodal Representation Learning by Alternating Unimodal Adaptationcited by 75
  11. 2023A Comprehensive Survey of Forgetting in Deep Learning Beyond Continual Learningcited by 69
  12. 2025Whoever Started the Interference Should End It: Guiding Data-Free Model Merging via Task Vectorscited by 62
38 more
  1. 2022Beyond Not-Forgetting: Continual Learning with Backward Knowledge Transfercited by 59
  2. 2021Recent Advances of Continual Learning in Computer Vision: An Overviewcited by 55
  3. 2021Powerpropagation: A sparsity inducing weight reparameterisationcited by 49
  4. 2021Natural continual learning: success is a journey, not (just) a destinationcited by 46
  5. 2021Adapting BERT for Continual Learning of a Sequence of Aspect Sentiment Classification Taskscited by 46
  6. 2022A Continual Deepfake Detection Benchmark: Dataset, Methods, and Essentialscited by 45
  7. 2022Memory Efficient Continual Learning with Transformerscited by 43
  8. 2023The Ideal Continual Learner: An Agent That Never Forgetscited by 35
  9. 2025FlyLoRA: Boosting Task Decoupling and Parameter Efficiency via Implicit Rank-Wise Mixture-of-Expertscited by 35
  10. 2021Towards Better Plasticity-Stability Trade-off in Incremental Learning: A Simple Linear Connectorcited by 34
  11. 2023Continual Learning with Scaled Gradient Projectioncited by 34
  12. 2022Continual Learning with Recursive Gradient Optimizationcited by 32
  13. 2024Visual Prompt Tuning in Null Space for Continual Learningcited by 31
  14. 2022Balancing Stability and Plasticity through Advanced Null Space in Continual Learningcited by 30
  15. 2021CLASSIC: Continual and Contrastive Learning of Aspect Sentiment Classification Taskscited by 29
  16. 2023Diversifying the Mixture-of-Experts Representation for Language Models with Orthogonal Optimizercited by 29
  17. 2023Learnability and Algorithm for Continual Learningcited by 28
  18. 2022A Multi-Head Model for Continual Learning via Out-of-Distribution Replaycited by 27
  19. 2023Dealing with Cross-Task Class Discrimination in Online Continual Learningcited by 24
  20. 2023Catastrophic Forgetting in Deep Learning: A Comprehensive Taxonomycited by 24
  21. 2022Lifelong Reinforcement Learning with Modulating Maskscited by 22
  22. 2023Open-World Continual Learning: Unifying Novelty Detection and Continual Learningcited by 22
  23. 2023Continual Learning in Medical Image Analysis: A Comprehensive Review of Recent Advancements and Future Prospectscited by 22
  24. 2024Multi-Task Model Merging via Adaptive Weight Disentanglementcited by 22
  25. 2022Continual Learning Based on OOD Detection and Task Maskingcited by 21
  26. 2023A Survey of Geometric Optimization for Deep Learning: From Euclidean Space to Riemannian Manifoldcited by 21
  27. 2023Regularizing Second-Order Influences for Continual Learningcited by 21
  28. 2022One-shot Federated Learning without Server-side Trainingcited by 20
  29. 2023Class Incremental Learning via Likelihood Ratio Based Task Predictioncited by 19
  30. 2022Discrete Key-Value Bottleneckcited by 18
  31. 2025LoRA Subtraction for Drift-Resistant Space in Exemplar-Free Continual Learningcited by 18
  32. 2021Cognitively Inspired Learning of Incremental Drifting Conceptscited by 16
  33. 2022A Theory for Knowledge Transfer in Continual Learningcited by 16
  34. 2023Two Complementary Perspectives to Continual Learning: Ask Not Only What to Optimize, But Also Howcited by 16
  35. 2025Upweighting Easy Samples in Fine-Tuning Mitigates Forgettingcited by 16
  36. 2023What to Remember: Self-Adaptive Continual Learning for Audio Deepfake Detectioncited by 15
  37. 2023FedSOL: Stabilized Orthogonal Learning with Proximal Restrictions in Federated Learningcited by 14
  38. 2024Learn to Preserve and Diversify: Parameter-Efficient Group with Orthogonal Regularization for Domain Generalizationcited by 14

50 most cited of 179 All 179 (JSON)