mlsanityData to 7 Oct 2026

Not How Many, But Which: Parameter Placement in Low-Rank Adaptation

arXiv 2605.12207 ↗First posted 12 May 2026cs.LG cs.AI cs.CLcited by 0

Not on the map. Why?

Abstract (arXiv)

We study the textitparameter placement problem: given a fixed budget of k trainable entries within the B matrix of a LoRA adapter (A frozen), does the choice of which k matter? Under supervised fine-tuning, random and informed subsets achieve comparable performance. Under GRPO on base models, random placement fails to improve over the base model, while gradient-informed placement recovers standard LoRA accuracy. This regime dependence traces to gradient structure: SFT gradients are low-rank and directionally stable, so any subset accumulates coherent updates; GRPO gradients are high-rank and near-orthogonal across steps, so only elements with consistently signed gradients retain the learning signal. Our scoring procedure identifies these critical parameters in under 10 seconds at less than 0.5% of training cost. Selected parameters concentrate on residual-stream-writing projections (V, O, Down), stable across model families and scales (1.5B - 8B).

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 70 of its 92 references that are in the map. First posted on arXiv 12 May 2026.

Builds on

70 of 92 references.

References · 70

  1. 2024The Llama 3 Herd of Modelscited by 14,192
  2. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  3. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 8,851
  4. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,916
  5. 2020Measuring Massive Multitask Language Understandingcited by 7,768
  6. 2025DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learningcited by 6,146
  7. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,203
  8. 2016Overcoming catastrophic forgetting in neural networkscited by 4,718
  9. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,710
  10. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,018
  11. 2023QLoRA: Efficient Finetuning of Quantized LLMscited by 3,355
  12. 2021TruthfulQA: Measuring How Models Mimic Human Falsehoodscited by 2,919
38 more
  1. 2019PIQA: Reasoning about Physical Commonsense in Natural Languagecited by 2,818
  2. 2023GPQA: A Graduate-Level Google-Proof Q&A Benchmarkcited by 2,748
  3. 2025DAPO: An Open-Source LLM Reinforcement Learning System at Scalecited by 2,678
  4. 2018The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networkscited by 2,250
  5. 2019BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questionscited by 2,140
  6. 2023MathVista: Evaluating Mathematical Reasoning of Foundation Models in Visual Contextscited by 1,582
  7. 2025Understanding R1-Zero-Like Training: A Critical Perspectivecited by 1,276
  8. 2021BitFit: Simple Parameter-efficient Fine-tuning for Transformer-based Masked Language-modelscited by 1,234
  9. 2022Interpretability in the Wild: a Circuit for Indirect Object Identification in GPT-2 smallcited by 1,130
  10. 2023SparseGPT: Massive Language Models Can Be Accurately Pruned in One-Shotcited by 1,090
  11. 2022Mass-Editing Memory in a Transformercited by 976
  12. 2023Progress measures for grokking via mechanistic interpretabilitycited by 822
  13. 2018SNIP: Single-shot Network Pruning based on Connection Sensitivitycited by 802
  14. 2023A Simple and Effective Pruning Approach for Large Language Modelscited by 776
  15. 2024DoRA: Weight-Decomposed Low-Rank Adaptationcited by 740
  16. 2020Intrinsic Dimensionality Explains the Effectiveness of Language Model Fine-Tuningcited by 658
  17. 2021Merging Models with Fisher-Weighted Averagingcited by 613
  18. 2023Eliciting Latent Predictions from Transformers with the Tuned Lenscited by 579
  19. 2020Pruning neural networks without any data by iteratively conserving synaptic flowcited by 475
  20. 2020Picking Winning Tickets Before Training by Preserving Gradient Flowcited by 467
  21. 2024GaLore: Memory-Efficient LLM Training by Gradient Low-Rank Projectioncited by 447
  22. 2020Movement Pruning: Adaptive Sparsity by Fine-Tuningcited by 398
  23. 2024LoRA+: Efficient Low Rank Adaptation of Large Modelscited by 365
  24. 2018Measuring the Intrinsic Dimension of Objective Landscapescited by 364
  25. 2020Parameter-Efficient Transfer Learning with Diff Pruningcited by 351
  26. 2024LoRA Learns Less and Forgets Lesscited by 313
  27. 2020The Lottery Ticket Hypothesis for Pre-trained BERT Networkscited by 275
  28. 2024PiSSA: Principal Singular Values and Singular Vectors Adaptation of Large Language Modelscited by 261
  29. 2023A Rank Stabilization Scaling Factor for Fine-Tuning with LoRAcited by 231
  30. 2022DyLoRA: Parameter Efficient Tuning of Pre-trained Models using Dynamic Search-Free Low-Rank Adaptationcited by 229
  31. 2023ReMax: A Simple, Effective, and Efficient Reinforcement Learning Method for Aligning Large Language Modelscited by 184
  32. 2023The Truth is in There: Improving Reasoning in Language Models with Layer-Selective Rank Reductioncited by 120
  33. 2024Flora: Low-Rank Adapters Are Secretly Gradient Compressorscited by 103
  34. 2023The Expressive Power of Low-Rank Adaptationcited by 100
  35. 2024AtP*: An efficient and scalable method for localizing LLM behaviour to componentscited by 100
  36. 2023Sparse Low-rank Adaptation of Pre-trained Language Modelscited by 96
  37. 2024LISA: Layerwise Importance Sampling for Memory-Efficient Large Language Model Fine-Tuningcited by 88
  38. 2022SparseAdapter: An Easy Approach for Improving the Parameter-Efficiency of Adapterscited by 83

50 most cited of 70 All 70 (JSON)

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026Rethinking Adapter Placement: A Dominant Adaptation Module Perspective
  2. 2026When Gradient Importance Lies: Adaptive LoRA Rank Allocation Fails Under GRPO
  3. 2024Flat-LoRA: Low-Rank Adaptation over a Flat Loss Landscape
  4. 2024LoRA-Pro: Are Low-Rank Adapters Properly Optimized?
  5. 2026Beyond Low-Rank Parameterization: Narrowing the Gap Between LoRA and Full Fine-Tuning via Gradient Decomposition
  6. 2025GraLoRA: Granular Low-Rank Adaptation for Parameter-Efficient Fine-Tuning
  7. 2025Revisiting LoRA through the Lens of Parameter Redundancy: Spectral Encoding Helps
  8. 2026Strategic Over-Parameterization for Generalizable Low-Rank Adaptation