mlsanityData to 14 Oct 2026

One-Step Curvature Probes Miss the Fitting Operator: Retained Capacity and Terminal Null-Space Correction for Continual Learning

arXiv 2610.03952 ↗First posted 2 Oct 2026cs.LGcited by 0

Not on the map. Why?

Abstract (arXiv)

A one-step curvature probe evaluates an initial direction, whereas continual learners are judged after reaching comparable new-task fit. In an overparameterized linearization, projected gradient descent converges to DeltaP=PJ^top(JPJ^top)-1r, and its squared-displacement inflation is exactly the reciprocal of the retained fitting capacity cP(r). More generally, the terminal old-task quadratic ratio factorizes as 1/[cP(r)Grm end(P,r)], where Grm end compares curvature along endpoint directions. In the rank-one case, Grm end equals the one-step probe gain; for multiple outputs, the two gains can differ. The terminal quadratic also separates into a curvature-optimal fitting floor and an algorithm-dependent null-space excess, motivating terminal null-space correction, which preserves linearized new-task outputs on its Jacobian batch. Controlled checks validate the local quadratic and show that projection can greatly reduce matched-norm curvature while barely changing terminal forgetting. Across common-threshold configurations, projection yields the larger signed old-task loss change in 73/99 matched pairs, retained fitting capacity falls with rank, and fixed-rank comparisons separate terminal forgetting even when probe gain is approximately matched. On Permuted MNIST and Split CIFAR-100, terminal correction decreases signed old-task loss in 166/180 method--dataset--seed pairs under the all-seed intention-to-correct analysis; because acceptance and outcome reporting use the same held-out split, this result is descriptive and test-conditioned. Overall, terminal cost depends jointly on retained fitting capacity, endpoint-direction curvature, and the null-space component selected by optimization.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 24 of its 38 references that are in the map. First posted on arXiv 2 Oct 2026.

Builds on

24 of 38 references.

References · 24

  1. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  2. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  3. 2017Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithmscited by 4,349
  4. 2017Understanding Black-box Predictions via Influence Functionscited by 1,741
  5. 2019Machine Unlearningcited by 1,061
  6. 2015Optimizing Neural Networks with Kronecker-factored Approximate Curvaturecited by 796
  7. 2019Eternal Sunshine of the Spotless Net: Selective Forgetting in Deep Networkscited by 618
  8. 2019Certified Data Removal from Machine Learning Modelscited by 535
  9. 2019Making AI Forget You: Data Deletion in Machine Learningcited by 465
  10. 2021Remember What You Want to Forget: Algorithms for Machine Unlearningcited by 371
  11. 2020Descent-to-Delete: Gradient-Based Methods for Machine Unlearningcited by 311
  12. 2020Approximate Data Deletion from Machine Learning Modelscited by 255
12 more
  1. 2016A Kronecker-factored approximate Fisher matrix for convolution layerscited by 141
  2. 2021Machine Unlearning via Algorithmic Stabilitycited by 130
  3. 2022Algorithms that Approximate Data Removal: New Results and Limitationscited by 40
  4. 2020Coded Machine Unlearningcited by 36
  5. 2024Towards Certified Unlearning for Deep Neural Networkscited by 31
  6. 2024Hessian-Free Online Certified Unlearningcited by 17
  7. 2023Ticketed Learning-Unlearning Schemescited by 11
  8. 2024Deep Unlearn: Benchmarking Machine Unlearning for Image Classificationcited by 9
  9. 2025Machine Unlearning via Information Theoretic Regularizationcited by 6
  10. 2025System-Aware Unlearning Algorithms: Use Lesser, Forget Fastercited by 4
  11. 2026Exact Federated Continual Unlearning for Ridge Heads on Frozen Foundation Modelscited by 4
  12. 2022The Complexity of Dynamic Least-Squares Regressioncited by 3

Built on by

Not cited yet.