mlsanityData to 7 Oct 2026

Efficient Parametric Approximations of Neural Network Function Space Distance

arXiv 2302.03519 ↗First posted 7 Feb 2023cs.LG cs.AI stat.MLcited by 6

Not on the map. Why?

Abstract (arXiv)

It is often useful to compactly summarize important properties of model parameters and training data so that they can be used later without storing and/or iterating over the entire dataset. As a specific case, we consider estimating the Function Space Distance (FSD) over a training set, i.e. the average discrepancy between the outputs of two neural networks. We propose a Linearized Activation Function TRick (LAFTR) and derive an efficient approximation to FSD for ReLU neural networks. The key idea is to approximate the architecture as a linear network with stochastic gating. Despite requiring only one parameter per unit of the network, our approach outcompetes other parametric approximations with larger memory requirements. Applied to continual learning, our parametric approximation is competitive with state-of-the-art nonparametric approximations, which require storing many training examples. Furthermore, we show its efficacy in estimating influence functions accurately and detecting mislabeled examples without expensive iterations over the entire dataset.

Key facts. Cited by 6 papers in the map (data through 7 Oct 2026). Builds on 37 of its 55 references that are in the map. First posted on arXiv 7 Feb 2023.

Builds on

37 of 55 references.

References · 37

  1. 2015Distilling the Knowledge in a Neural Networkcited by 10,076
  2. 2016Overcoming catastrophic forgetting in neural networkscited by 4,718
  3. 2018Neural Tangent Kernel: Convergence and Generalization in Neural Networkscited by 2,597
  4. 2017Gradient Episodic Memory for Continual Learningcited by 1,871
  5. 2017Understanding Black-box Predictions via Influence Functionscited by 1,714
  6. 2018Continual Lifelong Learning with Neural Networks: A Reviewcited by 1,313
  7. 2019Machine Unlearningcited by 1,020
  8. 2015Optimizing Neural Networks with Kronecker-factored Approximate Curvaturecited by 776
  9. 2013Exact solutions to the nonlinear dynamics of learning in deep linear neural networkscited by 748
  10. 2019Wide Neural Networks of Any Depth Evolve as Linear Models Under Gradient Descentcited by 732
  11. 2013An Empirical Investigation of Catastrophic Forgetting in Gradient-Based Neural Networkscited by 711
  12. 2021Fast Model Editing at Scalecited by 491
25 more
  1. 2017Variational Continual Learningcited by 335
  2. 2019SMART: Robust and Efficient Fine-Tuning for Pre-trained Natural Language Models through Principled Regularized Optimizationcited by 253
  3. 2019Limitations of the Empirical Fisher Approximation for Natural Gradient Descentcited by 213
  4. 2018Online Structured Laplace Approximations For Overcoming Catastrophic Forgettingcited by 197
  5. 2020Anatomy of Catastrophic Forgetting: Hidden Representations and Task Semanticscited by 158
  6. 2013Revisiting Natural Gradient for Deep Networkscited by 157
  7. 2022If Influence Functions are the Answer, Then What is the Question?cited by 142
  8. 2016A Kronecker-factored approximate Fisher matrix for convolution layerscited by 138
  9. 2021Scaling Up Influence Functionscited by 130
  10. 2019Functional Regularisation for Continual Learning with Gaussian Processescited by 121
  11. 2019Finite Depth and Width Corrections to the Neural Tangent Kernelcited by 114
  12. 2020Continual Deep Learning by Functional Regularisation of Memorable Pastcited by 104
  13. 2019Uncertainty-guided Continual Learning with Bayesian Neural Networkscited by 100
  14. 2020A Theoretical Analysis of Catastrophic Forgetting through the NTK Overlap Matrixcited by 99
  15. 2018Measuring and regularizing networks in function spacecited by 97
  16. 2019Dynamics of Deep Neural Networks and Neural Tangent Hierarchycited by 96
  17. 2021Wide Neural Networks Forget Less Catastrophicallycited by 64
  18. 2020On the distance between two neural networks and the stability of learningcited by 46
  19. 2020Analyzing Finite Neural Networks: Can We Trust Neural Tangent Kernel Theory?cited by 33
  20. 2022Neural Tangent Kernel Beyond the Infinite-Width Limit: Effects of Depth and Initializationcited by 30
  21. 2020Variational Auto-Regressive Gaussian Processes for Continual Learningcited by 23
  22. 2021Sequoia: A Software Framework to Unify Continual Learning Researchcited by 19
  23. 2021Task-agnostic Continual Learning with Hybrid Probabilistic Modelscited by 16
  24. 2020Taylorized Training: Towards Better Approximation of Neural Network Training at Finite Widthcited by 11
  25. 2022Amortized Proximal Optimizationcited by 9

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 2; 2024: 3; 2025: 0; 2026: 1.

6 papers.

  1. 2024Scalable Optimization in the Modular Normcited by 64
  2. 2024On the Diminishing Returns of Width for Continual Learningcited by 12
  3. 2023Preserving Linear Separability in Continual Learning by Backward Feature Projectioncited by 6
  4. 2024Streamlining Prediction in Bayesian Deep Learningcited by 5
  5. 2023Leveraging Function Space Aggregation for Federated Learning at Scalecited by 4
  6. 2026Dataless Weight Disentanglement in Task Arithmetic via Kronecker-Factored Approximate Curvaturecited by 3