mlsanityData to 7 Oct 2026

A Kernel-Based View of Language Model Fine-Tuning

arXiv 2210.05643 ↗First posted 11 Oct 2022cs.LG cs.CLcited by 118

Not on the map. Why?

Abstract (arXiv)

It has become standard to solve NLP tasks by fine-tuning pre-trained language models (LMs), especially in low-data settings. There is minimal theoretical understanding of empirical success, e.g., why fine-tuning a model with 108 or more parameters on a couple dozen training points does not result in overfitting. We investigate whether the Neural Tangent Kernel (NTK) - which originated as a model to study the gradient descent dynamics of infinitely wide networks with suitable random initialization - describes fine-tuning of pre-trained LMs. This study was inspired by the decent performance of NTK for computer vision tasks (Wei et al., 2022). We extend the NTK formalism to Adam and use Tensor Programs (Yang, 2020) to characterize conditions under which the NTK lens may describe fine-tuning updates to pre-trained language models. Extensive experiments on 14 NLP tasks validate our theory and show that formulating the downstream task as a masked word prediction problem through prompting often induces kernel-based dynamics during fine-tuning. Finally, we use this kernel view to propose an explanation for the success of parameter-efficient subspace-based fine-tuning methods.

Key facts. Cited by 118 papers in the map (data through 7 Oct 2026). Builds on 55 of its 90 references that are in the map. First posted on arXiv 11 Oct 2022.

Builds on

55 of 90 references.

References · 55

  1. 2017Attention Is All You Needcited by 59,932
  2. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  3. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  4. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  5. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,854
  6. 2016SQuAD: 100,000+ Questions for Machine Comprehension of Textcited by 4,115
  7. 2021Prefix-Tuning: Optimizing Continuous Prompts for Generationcited by 3,878
  8. 2021The Power of Scale for Parameter-Efficient Prompt Tuningcited by 3,769
  9. 2017A Broad-Coverage Challenge Corpus for Sentence Understanding through Inferencecited by 2,665
  10. 2015Character-level Convolutional Networks for Text Classificationcited by 2,293
  11. 2021Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processingcited by 2,222
  12. 2015A large annotated corpus for learning natural language inferencecited by 1,837
38 more
  1. 2020Making Pre-trained Language Models Better Few-shot Learnerscited by 1,255
  2. 2021BitFit: Simple Parameter-efficient Fine-tuning for Transformer-based Masked Language-modelscited by 1,234
  3. 2020Exploiting Cloze Questions for Few Shot Text Classification and Natural Language Inferencecited by 934
  4. 2019SpanBERT: Improving Pre-training by Representing and Predicting Spanscited by 832
  5. 2018A Convergence Theory for Deep Learning via Over-Parameterizationcited by 784
  6. 2020Intrinsic Dimensionality Explains the Effectiveness of Language Model Fine-Tuningcited by 658
  7. 2018Gradient Descent Provably Optimizes Over-parameterized Neural Networkscited by 655
  8. 2018Gradient Descent Finds Global Minima of Deep Neural Networkscited by 572
  9. 2019On Exact Computation with an Infinitely Wide Neural Netcited by 572
  10. 2019Fine-Grained Analysis of Optimization and Generalization for Overparameterized Two-Layer Neural Networkscited by 562
  11. 2019A Theoretical Analysis of Contrastive Unsupervised Representation Learningcited by 530
  12. 2018Measuring the Intrinsic Dimension of Objective Landscapescited by 364
  13. 2018Learning and Generalization in Overparameterized Neural Networks, Going Beyond Two Layerscited by 340
  14. 2020Kernel and Rich Regimes in Overparametrized Modelscited by 302
  15. 2022Tensor Programs V: Tuning Large Neural Networks via Zero-Shot Hyperparameter Transfercited by 272
  16. 2018Learning Overparameterized Neural Networks via Stochastic Gradient Descent on Structured Datacited by 260
  17. 2019Generalization Bounds of Stochastic Gradient Descent for Wide and Deep Neural Networkscited by 221
  18. 2020Few-Shot Learning via Learning the Representation, Provablycited by 213
  19. 2020Understanding the Difficulty of Training Transformerscited by 210
  20. 2020On the Theory of Transfer Learning: The Importance of Task Diversitycited by 186
  21. 2020Predicting What You Already Know Helps: Provable Self-Supervised Learningcited by 152
  22. 2019Tensor Programs I: Wide Feedforward or Recurrent Neural Networks of Any Architecture are Gaussian Processescited by 148
  23. 2020Contrastive learning, multi-view redundancy, and linear modelscited by 146
  24. 2020Tensor Programs II: Neural Tangent Kernel for Any Architecturecited by 139
  25. 2021Cutting Down on Prompts and Parameters: Simple Few-Shot Learning with Language Modelscited by 139
  26. 2020Understanding and Improving Information Transfer in Multi-Task Learningcited by 113
  27. 2021Exploring the Limits of Large Scale Pre-trainingcited by 104
  28. 2022On the SDEs and Scaling Rules for Adaptive Gradient Algorithmscited by 100
  29. 2022Understanding Contrastive Learning Requires Incorporating Inductive Biasescited by 96
  30. 2019Harnessing the Power of Infinitely Wide Deep Nets on Small-data Taskscited by 87
  31. 2022More Than a Toy: Random Matrix Models Predict How Real-World Neural Representations Generalizecited by 87
  32. 2020A Mathematical Exploration of Why Language Models Help Solve Downstream Taskscited by 71
  33. 2022Fast Finite Width Neural Tangent Kernelcited by 62
  34. 2021Tensor Programs IIb: Architectural Universality of Neural Tangent Kernel Training Dynamicscited by 57
  35. 2020Contrastive estimation reveals topic posterior information to linear modelscited by 54
  36. 2020Tensor Programs III: Neural Matrix Lawscited by 47
  37. 2021How Fine-Tuning Allows for Effective Meta-Learningcited by 41
  38. 2021Fast Adaptation with Linearized Neural Networkscited by 32

50 most cited of 55 All 55 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 2; 2023: 22; 2024: 31; 2025: 30; 2026: 33.

118 papers.

Sort:

By area:

  1. 2024LESS: Selecting Influential Data for Targeted Instruction Tuningcited by 514
  2. 2023Fine-Tuning Language Models with Just Forward Passescited by 337
  3. 2023TRAK: Attributing Model Behavior at Scalecited by 273
  4. 2023Task Arithmetic in the Tangent Space: Improved Editing of Pre-Trained Modelscited by 218
  5. 2023Scaling Down to Scale Up: A Guide to Parameter-Efficient Fine-Tuningcited by 171
  6. 2023What and How does In-Context Learning Learn? Bayesian Model Averaging, Parameterization, and Generalizationcited by 112
  7. 2023Mechanistically analyzing the effects of fine-tuning on procedurally defined taskscited by 110
  8. 2023Task-Specific Skill Localization in Fine-tuned Language Modelscited by 104
  9. 2023The Expressive Power of Low-Rank Adaptationcited by 100
  10. 2024A Survey on LoRA of Large Language Modelscited by 67
  11. 2024Unintentional Unalignment: Likelihood Displacement in Direct Preference Optimizationcited by 66
  12. 2024LoRA Training in the NTK Regime has No Spurious Local Minimacited by 48
38 more
  1. 2024Forcing Diffuse Distributions out of Language Modelscited by 48
  2. 2023Learning from models beyond fine-tuningcited by 47
  3. 2023The Journey, Not the Destination: How Data Guides Diffusion Modelscited by 42
  4. 2024Low-Rank Adaptation for Foundation Models: A Comprehensive Reviewcited by 35
  5. 2024Overcoming Reward Overoptimization via Adversarial Policy Optimization with Lightweight Uncertainty Estimationcited by 33
  6. 2023Universality and Limitations of Prompt Tuningcited by 30
  7. 2023Sparsity May Cry: Let Us Fail (Current) Sparse Neural Networks Together!cited by 27
  8. 2024Zeroth-Order Fine-Tuning of LLMs with Extreme Sparsitycited by 27
  9. 2025Tensor Product Attention Is All You Needcited by 27
  10. 2024Diffusion Soup: Model Merging for Text-to-Image Diffusion Modelscited by 25
  11. 2023ZO-AdaMU Optimizer: Adapting Perturbation by the Momentum and Uncertainty in Zeroth-order Optimizationcited by 24
  12. 2023GradientCoin: A Peer-to-Peer Decentralized Large Language Modelscited by 23
  13. 2024Understanding Linear Probing then Fine-tuning Language Models from NTK Perspectivecited by 21
  14. 2024Diffusion Attribution Score: Evaluating Training Data Influence in Diffusion Modelscited by 21
  15. 2026There Will Be a Scientific Theory of Deep Learningcited by 21
  16. 2024Scalable Multitask Learning Using Gradient-based Estimation of Task Affinitycited by 20
  17. 2024When does compositional structure yield compositional generalization? A kernel theorycited by 19
  18. 2025LoRA Training Provably Converges to a Low-Rank Global Minimum or It Fails Loudly (But it Probably Won't Fail)cited by 19
  19. 2023The Marginal Value of Momentum for Small Learning Rate SGDcited by 16
  20. 2024Random Masking Finds Winning Tickets for Parameter Efficient Fine-tuningcited by 16
  21. 2025Efficient Model Editing with Task-Localized Sparse Fine-tuningcited by 16
  22. 2025Mitigating Parameter Interference in Model Merging via Sharpness-Aware Fine-Tuningcited by 16
  23. 2022An Analysis of Attention via the Lens of Exchangeability and Latent Variable Modelscited by 15
  24. 2023Differentially Private Neural Tangent Kernels for Privacy-Preserving Data Generationcited by 15
  25. 2023Understanding Reconstruction Attacks with the Neural Tangent Kernel and Dataset Distillationcited by 14
  26. 2024Differential Privacy Mechanisms in Neural Tangent Kernel Regressioncited by 14
  27. 2024Helpful or Harmful Data? Fine-tuning-free Shapley Attribution for Explaining Language Model Predictionscited by 13
  28. 2024Scalable Fine-tuning from Multiple Data Sources: A First-Order Approximation Approachcited by 13
  29. 2025Understanding the Learning Dynamics of LoRA: A Gradient Flow Perspective on Low-Rank Adaptation in Matrix Factorizationcited by 13
  30. 2024Understanding Forgetting in LLM Supervised Fine-Tuning and Preference Learning -- A Convex Optimization Perspectivecited by 12
  31. 2023Unmasking Transformers: A Theoretical Approach to Data Recovery via Attention Weightscited by 11
  32. 2023Sample based Explanations via Generalized Representerscited by 10
  33. 2025Discrepancies are Virtue: Weak-to-Strong Generalization through Lens of Intrinsic Dimensioncited by 10
  34. 2025Parameter-Efficient Adaptation of Geospatial Foundation Models through Embedding Deflectioncited by 10
  35. 2024Learning Beyond Pattern Matching? Assaying Mathematical Understanding in LLMscited by 9
  36. 2025Representations Shape Weak-to-Strong Generalization: Theoretical Insights and Empirical Predictionscited by 9
  37. 2025ActiveDPO: Active Direct Preference Optimization for Sample-Efficient Alignmentcited by 9
  38. 2025Sharp Generalization Bounds for Foundation Models with Asymmetric Randomized Low-Rank Adapterscited by 9

50 most cited of 118 All 118 (JSON)