mlsanityData to 7 Oct 2026

How Much Rank Does LoRA Need? Rank-Error Bounds for Transformer Attention

arXiv 2608.26052 ↗First posted 26 Aug 2026cs.LG cs.AI cs.CLcited by 0

Not on the map. Why?

Abstract (arXiv)

Choosing the rank of a low-rank adaptation (LoRA) update is usually an empirical task. In this paper, we provide a task-dependent theory of the approximation error achievable at each LoRA rank for Transformer attention. We fix a pretrained attention head, a target attention function, and a distribution over inputs from the downstream task, and bound the smallest expected Kullback--Leibler (KL) error achievable by a rank-r query LoRA update. When target attention probabilities are bounded away from zero, we prove a lower bound of the error proportional to psi(|d|2), where d is the difference between candidate and target attention scores and psi(t)=mint2,t. We also prove an unconditional upper bound min|d|22/4,√2|d|2. Under explicit realizability, geometry, and moment conditions, we then bound the best rank-r error between an explicit multiple of psi(√Tr) and minTr/4,√2Tr, where Tr is the downstream-weighted tail energy of the target update. We also provide target-Fisher bounds when candidate scores remain within a fixed range of the target scores, and an unrestricted lower bound when a subset of tokens carries most of the probability mass. These spectral bounds describe finite-score approximation. We then construct explicit families in which softmax saturation makes the rank required to match the attention function strictly smaller than the rank required to match the finite logits. Finally, we extend the analysis to fused multi-head LoRA and joint query/key updates, exposing the effects of rank sharing and query/key factorization constraints.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 17 of its 31 references that are in the map. First posted on arXiv 26 Aug 2026.

Builds on

17 of 31 references.

References · 17

  1. 2017Attention Is All You Needcited by 59,932
  2. 2023AdaLoRA: Adaptive Budget Allocation for Parameter-Efficient Fine-Tuningcited by 337
  3. 2024SVD-LLM: Truncation-aware Singular Value Decomposition for Large Language Model Compressioncited by 231
  4. 2023The Expressive Power of Low-Rank Adaptationcited by 100
  5. 2026Kimi K3: Open Frontier Intelligencecited by 99
  6. 2020Low-Rank Bottleneck in Multi-head Attention Modelscited by 88
  7. 2024CorDA: Context-Oriented Decomposition Adaptation of Large Language Models for Task-Aware Parameter-Efficient Fine-tuningcited by 41
  8. 2024Implicit Geometry of Next-token Prediction: From Language Sparsity Patterns to Model Representationscited by 24
  9. 2019Globally Convergent Newton Methods for Ill-conditioned Generalized Self-concordant Lossescited by 17
  10. 2024Parameter Efficient Fine-tuning via Explained Variance Adaptationcited by 17
  11. 2022Low-Rank Softmax Can Have Unargmaxable Classes in Theory but Rarely in Practicecited by 14
  12. 2023Taming the Sigmoid Bottleneck: Provably Argmaxable Sparse Multi-Label Classificationcited by 11
5 more
  1. 2025Sequences of Logits Reveal the Low Rank Structure of Language Modelscited by 8
  2. 2025Unpacking Softmax: How Temperature Drives Representation Collapse, Compression, and Generalizationcited by 3
  3. 2026High-Dimensional Theory of LoRA Fine-Tuning in a Solvable Attention Modelcited by 3
  4. 2026Tight Sample Complexity for Low-Rank Adaptation: Matching Bounds and Rank Selectioncited by 2
  5. 2026The Entropic Bound for Transformers: Why Static Rank Fails and Attention-Native Rank Recoverscited by 1

Built on by

Not cited yet.