mlsanityData to 14 Oct 2026

High-Dimensional Theory of LoRA Fine-Tuning in a Solvable Attention Model

arXiv 2606.05899 ↗First posted 4 Jun 2026cs.LG cond-mat.dis-nncited by 3

Not on the map. Why?

Abstract (arXiv)

We develop a high-dimensional statistical theory of low-rank adaptation (LoRA) in attention models, capturing the interplay between pre-training and fine-tuning. We introduce a solvable framework in which a single-head attention layer is first pre-trained on a data-abundant task and subsequently adapted via a rank-one LoRA update on limited data. In the high-dimensional limit, both stages admit a sharp asymptotic characterization in terms of a finite set of order parameters, yielding explicit predictions for test errors and representation alignment. Our analysis shows that the impact of pre-training on LoRA is summarized by an effective noise term, from which we derive prescriptions for the optimal pre-training procedure. We also demonstrate a regime with a mismatch between the value of the test error and representation quality, and propose an application of our theory to active fine-tuning.

Key facts. Cited by 3 papers in the map (data through 14 Oct 2026). Builds on 33 of its 38 references that are in the map. First posted on arXiv 4 Jun 2026.

Builds on

33 of 38 references.

References · 33

  1. 2019Parameter-Efficient Transfer Learning for NLPcited by 4,236
  2. 2021Prefix-Tuning: Optimizing Continuous Prompts for Generationcited by 3,915
  3. 2023QLoRA: Efficient Finetuning of Quantized LLMscited by 3,418
  4. 2020Intrinsic Dimensionality Explains the Effectiveness of Language Model Fine-Tuningcited by 671
  5. 2023A Rank Stabilization Scaling Factor for Fine-Tuning with LoRAcited by 243
  6. 2020Online stochastic gradient descent on non-convex losses from high-dimensional inferencecited by 119
  7. 2023The Expressive Power of Low-Rank Adaptationcited by 102
  8. 2022Universality of empirical risk minimizationcited by 75
  9. 2023Smoothing the Landscape Boosts the Signal for SGD: Optimal Sample Complexity for Learning Single Index Modelscited by 61
  10. 2023Learning time-scales in two-layers neural networkscited by 57
  11. 2024LoRA Training in the NTK Regime has No Spurious Local Minimacited by 48
  12. 2025LoRA-One: One-Step Full Gradient Could Suffice for Fine-Tuning Large Language Models, Provably and Efficientlycited by 38
21 more
  1. 2024A phase transition between positional and semantic learning in a solvable model of dot-product attentioncited by 30
  2. 2023Universality laws for Gaussian mixtures in generalized linear modelscited by 28
  3. 2022An Equivalence Principle for the Spectrum of Random Inner-Product Kernel Matrices with Polynomial Scalingscited by 27
  4. 2025LoRA Training Provably Converges to a Low-Rank Global Minimum or It Fails Loudly (But it Probably Won't Fail)cited by 20
  5. 2025Asymptotics of Non-Convex Generalized Linear Models in High-Dimensions: A proof of the replica formulacited by 14
  6. 2025Understanding the Learning Dynamics of LoRA: A Gradient Flow Perspective on Low-Rank Adaptation in Matrix Factorizationcited by 13
  7. 2024High-dimensional learning of narrow neural networkscited by 12
  8. 2025Fundamental limits of learning in sequence multi-index models and deep attention networks: High-dimensional asymptotics and sharp thresholdscited by 12
  9. 2025Fundamental Limits of Matrix Sensing: Exact Asymptotics, Universality, and Applicationscited by 11
  10. 2025The Nuclear Route: Sharp Asymptotics of ERM in Overparameterized Quadratic Networkscited by 11
  11. 2025Sharp Generalization Bounds for Foundation Models with Asymmetric Randomized Low-Rank Adapterscited by 11
  12. 2025DeLoRA: Decoupling Angles and Strength in Low-rank Adaptationcited by 10
  13. 2025On the Convergence Rate of LoRA Gradient Descentcited by 10
  14. 2025Asymptotics of SGD in Sequence-Single Index Models and Single-Layer Attention Networkscited by 9
  15. 2026A solvable high-dimensional model where nonlinear autoencoders learn structure invisible to PCA while test loss misaligns with generalizationcited by 8
  16. 2025Statistical Advantage of Softmax Attention: Insights from Single-Location Regressioncited by 7
  17. 2025Single-Head Attention in High Dimensions: A Theory of Generalization, Weights Spectra, and Scaling Lawscited by 6
  18. 2025Does Low Rank Adaptation Lead to Lower Robustness against Training-Time Attacks?cited by 4
  19. 2025Bayes optimal learning of attention-indexed modelscited by 3
  20. 2026Why LoRA Resists Label Noise: A Theoretical Framework for Noise-Robust Parameter-Efficient Fine-Tuningcited by 3
  21. 2026When pre-training hurts LoRA fine-tuning: a dynamical analysis via single-index modelscited by 2

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 3.

3 papers.

  1. 2026The Intruder Threshold: A Spectral Law for LoRA Fine-Tuningcited by 2
  2. 2026How Much Rank Does LoRA Need? Rank-Error Bounds for Transformer Attentioncited by 0
  3. 2026A Dynamical Theory of LoRA in Continual Learningcited by 0