mlsanityData to 13 Oct 2026

Fast Adaptation with Linearized Neural Networks

arXiv 2103.01439 ↗First posted 2 Mar 2021stat.ML cs.LGcited by 32

Not on the map. Why?

Abstract (arXiv)

The inductive biases of trained neural networks are difficult to understand and, consequently, to adapt to new settings. We study the inductive biases of linearizations of neural networks, which we show to be surprisingly good summaries of the full network functions. Inspired by this finding, we propose a technique for embedding these inductive biases into Gaussian processes through a kernel designed from the Jacobian of the network. In this setting, domain adaptation takes the form of interpretable posterior inference, with accompanying uncertainty estimation. This inference is analytic and free of local optima issues found in standard techniques such as fine-tuning neural network weights to a new task. We develop significant computational speed-ups based on matrix multiplies, including a novel implementation for scalable Fisher vector products. Our experiments on both image classification and regression demonstrate the promise and convenience of this framework for transfer learning, compared to neural network fine-tuning. Code is available at https://github.com/amzn/xfer/tree/master/finite_ntk.

Key facts. Cited by 32 papers in the map (data through 13 Oct 2026). Builds on 27 of its 51 references that are in the map. First posted on arXiv 2 Mar 2021.

Builds on

27 of 51 references.

References · 27

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,950
  2. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,223
  3. 2016Densely Connected Convolutional Networkscited by 6,707
  4. 2017On Calibration of Modern Neural Networkscited by 4,902
  5. 2017Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networkscited by 4,870
  6. 2017Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithmscited by 4,328
  7. 2014How transferable are features in deep neural networks?cited by 1,219
  8. 2019Wide Neural Networks of Any Depth Evolve as Linear Models Under Gradient Descentcited by 737
  9. 2018GPyTorch: Blackbox Matrix-Matrix Gaussian Process Inference with GPU Accelerationcited by 605
  10. 2019On Exact Computation with an Infinitely Wide Neural Netcited by 573
  11. 2015Deep Kernel Learningcited by 441
  12. 2016Adversarial Feature Learningcited by 350
15 more
  1. 2014CNN Features off-the-shelf: an Astounding Baseline for Recognitioncited by 316
  2. 2015Convergent Learning: Do different neural networks learn the same representations?cited by 215
  3. 2019Task2Vec: Task Embedding for Meta-Learningcited by 201
  4. 2018Bayesian Model-Agnostic Meta-Learningcited by 194
  5. 2019Scaling Limits of Wide Neural Networks with Weight Sharing: Gaussian Process Behavior, Gradient Independence, and Neural Tangent Kernel Derivationcited by 148
  6. 2020Continual Deep Learning by Functional Regularisation of Memorable Pastcited by 104
  7. 2019Approximate Inference Turns Deep Networks into Gaussian Processescited by 89
  8. 2019Enhanced Convolutional Neural Tangent Kernelscited by 76
  9. 2018Constant-Time Predictive Distributions for Gaussian Processescited by 48
  10. 2019BoTorch: A Framework for Efficient Monte-Carlo Bayesian Optimizationcited by 30
  11. 2020Gradients as Features for Deep Representation Learningcited by 30
  12. 2014Metrics for Probabilistic Geometriescited by 28
  13. 2019Adaptive Deep Kernel Learningcited by 15
  14. 2019Bayesian Meta-Learning for the Few-Shot Setting via Deep Kernelscited by 6
  15. 2019Adversarial Fisher Vectors for Unsupervised Representation Learningcited by 5

Built on by

Cited by year (2026 to 13 Oct)

Citing papers per year: 2021: 3; 2022: 13; 2023: 8; 2024: 5; 2025: 2; 2026: 1.

32 papers.

Sort:

  1. 2022Last Layer Re-Training is Sufficient for Robustness to Spurious Correlationscited by 380
  2. 2023TRAK: Attributing Model Behavior at Scalecited by 279
  3. 2023Task Arithmetic in the Tangent Space: Improved Editing of Pre-Trained Modelscited by 219
  4. 2022A Kernel-Based View of Language Model Fine-Tuningcited by 119
  5. 2021Priors in Bayesian Deep Learning: A Reviewcited by 101
  6. 2022Head2Toe: Utilizing Intermediate Representations for Better Transfer Learningcited by 83
  7. 2021Personalized Federated Learning with Gaussian Processescited by 76
  8. 2021What can linearized neural networks actually say about generalization?cited by 49
  9. 2022Pre-Train Your Loss: Easy Bayesian Transfer Learning with Informative Priorscited by 43
  10. 2023The Tunnel Effect: Building Data Representations in Deep Neural Networkscited by 34
  11. 2023Making Substitute Models More Bayesian Can Enhance Transferability of Adversarial Examplescited by 32
  12. 2022Adapting the Linearised Laplace Model Evidence for Modern Deep Learningcited by 29
20 more
  1. 2023Differentially Private Image Classification by Learning Priors from Random Processescited by 19
  2. 2022Demystifying the Neural Tangent Kernel from a Practical Perspective: Can it be trusted for Neural Architecture Search without training?cited by 18
  3. 2022Sampling-based inference for large linear models, with application to linearised Laplacecited by 18
  4. 2022On the adaptation of recurrent neural networks for system identificationcited by 13
  5. 2022Uncertainty Estimation for Computed Tomography with a Linearised Deep Image Priorcited by 10
  6. 2022Few-Shot Learning by Dimensionality Reduction in Gradient Spacecited by 8
  7. 2022Scale-invariant Bayesian Neural Networks with Connectivity Tangent Kernelcited by 7
  8. 2022Tractable Uncertainty-Aware Meta-Learningcited by 4
  9. 2022Cold Posteriors through PAC-Bayescited by 3
  10. 2023Improving Transferability of Adversarial Examples via Bayesian Attackscited by 3
  11. 2023The fine print on tempered posteriorscited by 3
  12. 2024Fine-tuning with Very Large Dropoutcited by 3
  13. 2024Regularized KL-Divergence for Well-Defined Function-Space Variational Inference in Bayesian neural networkscited by 3
  14. 2023Feature Expansion for Graph Neural Networkscited by 2
  15. 2024SGD with memory: fundamental properties and stochastic accelerationcited by 2
  16. 2025CLAPS: Aleatoric-Epistemic Scaling via Last-Layer Laplace for Conformal Regressioncited by 2
  17. 2024Generalization error of spectral algorithmscited by 1
  18. 2025Approximating Full Conformal Prediction for Neural Network Regression with Gauss-Newton Influencecited by 1
  19. 2026Is the Last Layer Sufficient for Uncertainty Quantification?cited by 1
  20. 2024Dropout MPC: An Ensemble Neural MPC Approach for Systems with Learned Dynamicscited by 0