mlsanityData to 14 Oct 2026

Knowledge Adaptation as Posterior Correction

arXiv 2506.14262 ↗First posted 17 Jun 2025cs.LG cs.AI stat.MLcited by 6

Not on the map. Why?

Abstract (arXiv)

Adaptation is the holy grail of intelligence, but even the best AI models lack the adaptability of toddlers. In spite of great progress, little is known about the mechanisms by which machines can learn to adapt as fast as humans and animals. Here, we cast adaptation as `correction' of old posteriors and show that a wide-variety of existing adaptation methods follow this very principle, including those used for continual learning, federated learning, unlearning, and model merging. In all these settings, more accurate posteriors often lead to smaller corrections and can enable faster adaptation. Posterior correction is derived by using the dual representation of the Bayesian Learning Rule of Khan and Rue (2023), where the interference between the old representation and new information is quantified by using the natural-gradient mismatch. We present many examples demonstrating how machines can learn to adapt quickly by using posterior correction.

Key facts. Cited by 6 papers in the map (data through 14 Oct 2026). Builds on 43 of its 79 references that are in the map. First posted on arXiv 17 Jun 2025.

Builds on

43 of 79 references.

References · 43

  1. 2015Distilling the Knowledge in a Neural Networkcited by 10,324
  2. 2016Communication-Efficient Learning of Deep Networks from Decentralized Datacited by 7,669
  3. 2016Overcoming catastrophic forgetting in neural networkscited by 4,787
  4. 2016iCaRL: Incremental Classifier and Representation Learningcited by 2,333
  5. 2017Gradient Episodic Memory for Continual Learningcited by 1,898
  6. 2017Understanding Black-box Predictions via Influence Functionscited by 1,741
  7. 2022Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference timecited by 1,396
  8. 2022Editing Models with Task Arithmeticcited by 1,120
  9. 2020Dark Experience for General Continual Learning: a Strong, Simple Baselinecited by 911
  10. 2021Robust fine-tuning of zero-shot modelscited by 797
  11. 2020Estimating Training Data Influence by Tracing Gradient Descentcited by 580
  12. 2021Federated Learning Based on Dynamic Regularizationcited by 562
31 more
  1. 2021Fast Model Editing at Scalecited by 498
  2. 2014New insights and perspectives on the natural gradient methodcited by 443
  3. 2017Variational Continual Learningcited by 339
  4. 2016Structured Inference Networks for Nonlinear State Space Modelscited by 182
  5. 2013Expectation Propagation for approximate Bayesian inferencecited by 174
  6. 2020Continual Deep Learning by Functional Regularisation of Memorable Pastcited by 105
  7. 2018Measuring and regularizing networks in function spacecited by 99
  8. 2021The Bayesian Learning Rulecited by 99
  9. 2019Approximate Inference Turns Deep Networks into Gaussian Processescited by 89
  10. 2023Model Merging by Uncertainty-Based Gradient Matchingcited by 77
  11. 2017Conjugate-Computation Variational Inference : Converting Variational Inference in Non-Conjugate Models to Inferences in Conjugate Modelscited by 65
  12. 2018Task Agnostic Continual Learning Using Online Variational Bayescited by 61
  13. 2013Streaming Variational Bayescited by 48
  14. 2022Federated Learning via Inexact ADMMcited by 44
  15. 2017Streaming Sparse Gaussian Process Approximationscited by 41
  16. 2018Fast yet Simple Natural-Gradient Descent for Variational Inference in Complex Modelscited by 38
  17. 2022FedADMM: A Federated Primal-Dual Algorithm Allowing Partial Participationcited by 36
  18. 2015Black box variational inference for state space modelscited by 31
  19. 2022FedADMM: A Robust Federated Deep Learning Framework with Adaptivity to System Heterogeneitycited by 25
  20. 2021Knowledge-Adaptation Priorscited by 20
  21. 2021Online Variational Filtering and Parameter Learningcited by 19
  22. 2021Dual Parameterization of Sparse Variational Gaussian Processescited by 19
  23. 2019A Generalization Bound for Online Variational Inferencecited by 17
  24. 2023The Memory Perturbation Equation: Understanding Model's Sensitivity to Datacited by 16
  25. 2014Expectation propagation as a way of life: A framework for Bayesian inference on partitioned datacited by 14
  26. 2024Bayesian Online Natural Gradient (BONG)cited by 14
  27. 2022Partitioned Variational Inference: A Framework for Probabilistic Federated Learningcited by 13
  28. 2023Memory-Based Dual Gaussian Processes for Sequential Learningcited by 11
  29. 2025Connecting Federated ADMM to Bayescited by 4
  30. 2022Bayesian data fusion with shared priorscited by 3
  31. 2025Federated ADMM from Bayesian Dualitycited by 3

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 1; 2026: 5.

6 papers.

  1. 2026Position: agentic AI orchestration should be Bayes-consistentcited by 3
  2. 2026SOAP-Bubbles: Structured Weight Uncertainty for Neural Networkscited by 3
  3. 2025SVRG and Beyond via Posterior Correctioncited by 0
  4. 2026Joint Model and Data Sparsification via the Marginal Likelihoodcited by 0
  5. 2026Fast and Slow Variational Continual Learningcited by 0
  6. 2026A Generalization of Amari's Bayesian Dualitycited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Knowledge is Not Enough: Injecting RL Skills for Continual Adaptation
  2. 2025Continual Knowledge Adaptation for Reinforcement Learning
  3. 2026A Six-Dimensional Taxonomy of Post-Training Adaptation Techniques with Applications in AI Governance
  4. 2026Knowledge Interaction in Hypernetwork-based Continual Learning: Conflict and Composition
  5. 2026DIMoE-Adapters: Dynamic Expert Evolution for Continual Learning in Vision-Language Models
  6. 2026Spectral Imbalance Causes Forgetting in Low-Rank Continual Adaptation
  7. 2023TAIL: Task-specific Adapters for Imitation Learning with Large Pretrained Models
  8. 2026Filtering Beats Fine Tuning: A Bayesian Kalman View of In Context Learning in LLMs