mlsanityData to 14 Oct 2026

Knowledge-Adaptation Priors

arXiv 2106.08769 ↗First posted 16 Jun 2021cs.LG cs.AI stat.MLcited by 20

Not on the map. Why?

Abstract (arXiv)

Humans and animals have a natural ability to quickly adapt to their surroundings, but machine-learning models, when subjected to changes, often require a complete retraining from scratch. We present Knowledge-adaptation priors (K-priors) to reduce the cost of retraining by enabling quick and accurate adaptation for a wide-variety of tasks and models. This is made possible by a combination of weight and function-space priors to reconstruct the gradients of the past, which recovers and generalizes many existing, but seemingly-unrelated, adaptation strategies. Training with simple first-order gradient methods can often recover the exact retrained model to an arbitrary accuracy by choosing a sufficiently large memory of the past data. Empirical results show that adaptation with K-priors achieves performance similar to full retraining, but only requires training on a handful of past examples.

Key facts. Cited by 20 papers in the map (data through 14 Oct 2026). Builds on 32 of its 75 references that are in the map. First posted on arXiv 16 Jun 2021.

Builds on

32 of 75 references.

References · 32

  1. 2014Adam: A Method for Stochastic Optimizationcited by 37,240
  2. 2015Distilling the Knowledge in a Neural Networkcited by 10,324
  3. 2016Overcoming catastrophic forgetting in neural networkscited by 4,787
  4. 2016Learning without Forgettingcited by 2,496
  5. 2016iCaRL: Incremental Classifier and Representation Learningcited by 2,333
  6. 2017Gradient Episodic Memory for Continual Learningcited by 1,898
  7. 2020Dark Experience for General Continual Learning: a Strong, Simple Baselinecited by 911
  8. 2019Eternal Sunshine of the Spotless Net: Selective Forgetting in Deep Networkscited by 618
  9. 2019Gradient based sample selection for online continual learningcited by 571
  10. 2019Certified Data Removal from Machine Learning Modelscited by 535
  11. 2018Progress & Compress: A scalable framework for continual learningcited by 520
  12. 2019Making AI Forget You: Data Deletion in Machine Learningcited by 465
20 more
  1. 2014New insights and perspectives on the natural gradient methodcited by 443
  2. 2019Online Continual Learning with Maximally Interfered Retrievalcited by 379
  3. 2017Variational Continual Learningcited by 339
  4. 2017Overcoming Catastrophic Forgetting by Incremental Moment Matchingcited by 243
  5. 2015Unifying distillation and privileged informationcited by 233
  6. 2018Online Structured Laplace Approximations For Overcoming Catastrophic Forgettingcited by 201
  7. 2020Challenges in Deploying Machine Learning: a Survey of Case Studiescited by 152
  8. 2020Machine Unlearning for Random Forestscited by 131
  9. 2019Functional Regularisation for Continual Learning with Gaussian Processescited by 123
  10. 2020Variational Bayesian Unlearningcited by 117
  11. 2020Continual Deep Learning by Functional Regularisation of Memorable Pastcited by 105
  12. 2018Measuring and regularizing networks in function spacecited by 99
  13. 2019Continual Learning in Practicecited by 28
  14. 2014Fast Randomized Kernel Methods With Statistical Guaranteescited by 21
  15. 2017Knowledge Adaptation: Teaching to Adaptcited by 17
  16. 2020Predictive Complexity Priorscited by 14
  17. 2014Penalising model component complexity: A principled, practical approach to constructing priorscited by 13
  18. 2018Infinite-Horizon Gaussian Processescited by 6
  19. 2017Adaptive SVM+: Learning with Privileged Information for Domain Adaptationcited by 5
  20. 2016Improving Efficiency of SVM k-fold Cross-validation by Alpha Seedingcited by 1

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2022: 4; 2023: 6; 2024: 5; 2025: 4; 2026: 1.

20 papers.

Sort:

By area:

  1. 2022A Survey of Machine Unlearningcited by 312
  2. 2023KGA: A General Machine Unlearning Framework Based on Knowledge Gap Alignmentcited by 99
  3. 2023Model Merging by Uncertainty-Based Gradient Matchingcited by 77
  4. 2024Machine Unlearning: A Comprehensive Surveycited by 35
  5. 2023A Primer on Bayesian Neural Networks: Review and Debatescited by 27
  6. 2023Low-rank extended Kalman filtering for online learning of neural networks from streaming datacited by 23
  7. 2023Self-Expanding Neural Networkscited by 14
  8. 2024On the Diminishing Returns of Width for Continual Learningcited by 12
  9. 2023Memory-Based Dual Gaussian Processes for Sequential Learningcited by 11
  10. 2024FSP-Laplace: Function-Space Priors for the Laplace Approximation in Bayesian Deep Learningcited by 11
  11. 2022Perspectives on Incorporating Expert Feedback into Model Updatescited by 7
  12. 2025Knowledge Adaptation as Posterior Correctioncited by 6
8 more
  1. 2024GUIDE: Guidance-based Incremental Learning with Diffusion Modelscited by 5
  2. 2025Connecting Federated ADMM to Bayescited by 4
  3. 2022Laplace HypoPINN: Physics-Informed Neural Network for hypocenter localization and its predictive uncertaintycited by 3
  4. 2022Challenges and Pitfalls of Bayesian Unlearningcited by 3
  5. 2025Improvements of Dark Experience Replay and Reservoir Sampling towards Better Balance between Consolidation and Plasticitycited by 2
  6. 2025Compact Memory for Continual Logistic Regressioncited by 1
  7. 2024BOWL: A Deceptively Simple Open World Learnercited by 0
  8. 2026Online Bayesian Node Classification on Inductive Graphs under Distribution Shiftcited by 0