mlsanityData to 7 Oct 2026

When Can Prefixes Compile LoRA? Exact Resource-Capped Tests for Frozen Attention

arXiv 2609.36766 ↗First posted 29 Sep 2026cs.LGcited by 0

Not on the map. Why?

Abstract (arXiv)

Can a fixed continuous prefix replace a given low-rank adapter while the attention head stays frozen? In this research, we show that the answer depends on the adapter's target through three conditions. First, observability: at one causal readout, every independent key--value prefix sees the content only through the query, attention partition, and value numerator, so a target that differs on two inputs with equal summaries incurs an error floor at every prefix length; norm caps extend this floor to nearly equal summaries. Second, realizability: at a common query, any prefix reduces exactly to two aggregate variables, and the norm-capped optimum is an attained second-order-cone program, also after a fixed output projection; it places two equal-norm rank-one value updates on opposite sides of compilability. Third, implementation: under affine query exposure, 2r signed slots approximate a rank-r value update, but their values grow as O(ε-3/2), and the construction passes all 400 tolerance checks in float64 yet only 38 in bfloat16. A first-layer GPT-2 readout with fixed token and position meets the common-query condition without clamping activations; at three such heads, the capped optimum leaves 18.4% to 74.2% of the projected adapter effect uncompiled, with a head-dependent value--query ordering. All claims concern local approximation at one head, not whole-network equivalence.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 43 of its 50 references that are in the map. First posted on arXiv 29 Sep 2026.

Builds on

43 of 50 references.

References · 43

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,160
  3. 2019Parameter-Efficient Transfer Learning for NLPcited by 4,195
  4. 2021Prefix-Tuning: Optimizing Continuous Prompts for Generationcited by 3,878
  5. 2021The Power of Scale for Parameter-Efficient Prompt Tuningcited by 3,769
  6. 2021Convex optimizationcited by 2,479
  7. 2022Rethinking the Role of Demonstrations: What Makes In-Context Learning Work?cited by 1,413
  8. 2021What Makes Good In-Context Examples for GPT-3?cited by 1,184
  9. 2022In-context Learning and Induction Headscited by 930
  10. 2021An Explanation of In-context Learning as Implicit Bayesian Inferencecited by 915
  11. 2022Few-Shot Parameter-Efficient Fine-Tuning is Better and Cheaper than In-Context Learningcited by 886
  12. 2024DoRA: Weight-Decomposed Low-Rank Adaptationcited by 740
31 more
  1. 2022Transformers learn in-context by gradient descentcited by 714
  2. 2022What Can Transformers Learn In-Context? A Case Study of Simple Function Classescited by 700
  3. 2022What learning algorithm is in-context learning? Investigations with linear modelscited by 671
  4. 2024Parameter-Efficient Fine-Tuning for Large Models: A Comprehensive Surveycited by 641
  5. 2023In-Context Learning Creates Task Vectorscited by 302
  6. 2023Function Vectors in Large Language Modelscited by 292
  7. 2023Transformers as Statisticians: Provable In-Context Learning with In-Context Algorithm Selectioncited by 291
  8. 2023Transformers learn to implement preconditioned gradient descent for in-context learningcited by 274
  9. 2023Transformers as Algorithms: Generalization and Stability in In-context Learningcited by 220
  10. 2020The Lipschitz Constant of Self-Attentioncited by 171
  11. 2023Scaling Down to Scale Up: A Guide to Parameter-Efficient Fine-Tuningcited by 171
  12. 2023One Step of Gradient Descent is Provably the Optimal In-Context Learner with One Layer of Linear Self-Attentioncited by 168
  13. 2023What In-Context Learning "Learns" In-Context: Disentangling Task Recognition and Task Learningcited by 149
  14. 2023The Learnability of In-Context Learningcited by 126
  15. 2023The Expressive Power of Low-Rank Adaptationcited by 100
  16. 2023When Do Prompting and Prefix-Tuning Work? A Theory of Capabilities and Limitationscited by 41
  17. 2025Learning without training: The implicit dynamics of in-context learningcited by 41
  18. 2023Universality and Limitations of Prompt Tuningcited by 30
  19. 2024Prompting a Pretrained Transformer Can Be a Universal Approximatorcited by 25
  20. 2026SHINE: A Scalable In-Context Hypernetwork for Mapping Context to LoRA in a Single Passcited by 22
  21. 2024Fundamental Limits of Prompt Tuning Transformers: Universality, Capacity and Efficiencycited by 21
  22. 2024Deeper Insights Without Updates: The Power of In-Context Learning Over Fine-Tuningcited by 20
  23. 2026Latent Context Compilation: Distilling Long Context into Compact Portable Memorycited by 11
  24. 2024Exact Conversion of In-Context Learning to Model Weights in Linearized-Attention Transformerscited by 6
  25. 2025A Theoretical Framework for Prompt Engineering: Approximating Smooth Functions with Transformer Promptscited by 6
  26. 2025Adaptive Task Vectors for Large Language Modelscited by 4
  27. 2025PrefixMemory-Tuning: Modernizing Prefix-Tuning by Decoupling the Prefix from Attentioncited by 4
  28. 2025Memory Limitations of Prompt Tuning in Transformerscited by 4
  29. 2025Transmuting prompts into weightscited by 3
  30. 2026ReasonCACHE: Teaching LLMs To Reason Without Weight Updatescited by 3
  31. 2026Training-Free Universal Approximation by Prompting Random Transformerscited by 1

Built on by

Not cited yet.