mlsanityData to 7 Oct 2026

Revisiting LoRA through the Lens of Parameter Redundancy: Spectral Encoding Helps

arXiv 2506.16787 ↗First posted 20 Jun 2025cs.LGcited by 2

Not on the map. Why?

Abstract (arXiv)

Low-Rank Adaptation (LoRA) has emerged as a prominent technique for fine-tuning large foundation models. Despite its successes, the substantial parameter redundancy, which limits the capacity and efficiency of LoRA, has been recognized as a bottleneck. In this work, we systematically investigate the impact of redundancy in fine-tuning LoRA and reveal that reducing density redundancy does not degrade expressiveness. Based on this insight, we introduce underlineSpectral-underlineencoding underlineLow-underlineRank underlineAdaptation (SeLoRA), which harnesses the robust expressiveness of spectral bases to re-parameterize LoRA from a sparse spectral subspace. Designed with simplicity, SeLoRA enables seamless integration with various LoRA variants for performance boosting, serving as a scalable plug-and-play framework. Extensive experiments substantiate that SeLoRA achieves greater efficiency with fewer parameters, delivering superior performance enhancements over strong baselines on various downstream tasks, including commonsense reasoning, math reasoning, and code generation.

Key facts. Cited by 2 papers in the map (data through 7 Oct 2026). Builds on 55 of its 80 references that are in the map. First posted on arXiv 20 Jun 2025.

Builds on

55 of 80 references.

References · 55

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2024The Llama 3 Herd of Modelscited by 14,192
  4. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  5. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  6. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  7. 2023Segment Anythingcited by 8,158
  8. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  9. 2020Measuring Massive Multitask Language Understandingcited by 7,768
  10. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,710
  11. 2019Parameter-Efficient Transfer Learning for NLPcited by 4,195
  12. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,018
38 more
  1. 2021Prefix-Tuning: Optimizing Continuous Prompts for Generationcited by 3,878
  2. 2021The Power of Scale for Parameter-Efficient Prompt Tuningcited by 3,769
  3. 2015Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classificationcited by 3,590
  4. 2021Program Synthesis with Large Language Modelscited by 3,509
  5. 2019PIQA: Reasoning about Physical Commonsense in Natural Languagecited by 2,818
  6. 2015Deep Compression: Compressing Deep Neural Networks with Pruning, Trained Quantization and Huffman Codingcited by 2,291
  7. 2018The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networkscited by 2,250
  8. 2019BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questionscited by 2,140
  9. 2018Can a Suit of Armor Conduct Electricity? A New Dataset for Open Book Question Answeringcited by 2,114
  10. 2015Learning both Weights and Connections for Efficient Neural Networkscited by 1,998
  11. 2023Is Your Code Generated by ChatGPT Really Correct? Rigorous Evaluation of Large Language Models for Code Generationcited by 1,332
  12. 2023SparseGPT: Massive Language Models Can Be Accurately Pruned in One-Shotcited by 1,090
  13. 2021Towards a Unified View of Parameter-Efficient Transfer Learningcited by 863
  14. 2018SNIP: Single-shot Network Pruning based on Connection Sensitivitycited by 802
  15. 2020AdapterFusion: Non-Destructive Task Composition for Transfer Learningcited by 780
  16. 2023WizardCoder: Empowering Code Large Language Models with Evol-Instructcited by 728
  17. 2021P-Tuning v2: Prompt Tuning Can Be Comparable to Fine-tuning Universally Across Scales and Taskscited by 686
  18. 2023Language Models are Super Mario: Absorbing Abilities from Homologous Models as a Free Lunchcited by 679
  19. 2020Intrinsic Dimensionality Explains the Effectiveness of Language Model Fine-Tuningcited by 658
  20. 2020Picking Winning Tickets Before Training by Preserving Gradient Flowcited by 467
  21. 2021Compacter: Efficient Low-Rank Hypercomplex Adapter Layerscited by 458
  22. 2024ShortGPT: Layers in Large Language Models are More Redundant Than You Expectcited by 369
  23. 2017Scalable Training of Artificial Neural Networks with Adaptive Sparse Connectivity inspired by Network Sciencecited by 325
  24. 2024LoRA Learns Less and Forgets Lesscited by 313
  25. 2023VeRA: Vector-based Random Matrix Adaptationcited by 294
  26. 2023Magicoder: Empowering Code Generation with OSS-Instructcited by 251
  27. 2023LoRA-FA: Efficient and Effective Low Rank Representation Fine-tuningcited by 132
  28. 2022Understanding and Improving Visual Prompting: A Label-Mapping Perspectivecited by 86
  29. 2022The Unreasonable Effectiveness of Random Pruning: Return of the Most Naive Baseline for Sparse Trainingcited by 85
  30. 2022SparseAdapter: An Easy Approach for Improving the Parameter-Efficiency of Adapterscited by 83
  31. 2024What Matters in Transformers? Not All Attention is Neededcited by 83
  32. 2023DeepZero: Scaling up Zeroth-Order Optimization for Deep Model Trainingcited by 76
  33. 2024LoRA-XS: Low-Rank Adaptation with Extremely Small Number of Parameterscited by 73
  34. 2024VB-LoRA: Extreme Parameter Efficient Fine-Tuning with Vector Bankscited by 52
  35. 2024RoSA: Accurate Parameter-Efficient Fine-Tuning via Robust Adaptationcited by 48
  36. 2021Leveraging redundancy in attention with Reuse Transformerscited by 33
  37. 2024APT: Adaptive Pruning and Tuning Pretrained Language Models for Efficient Training and Inferencecited by 29
  38. 2025G-Refer: Graph Retrieval-Augmented Large Language Model for Explainable Recommendationcited by 26

50 most cited of 55 All 55 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 2.

2 papers.

  1. 2026Sparse Spectral LoRA: Routed Experts for Medical VLMscited by 1
  2. 2026Let Triggers Control: Frequency-Aware Dropout for Effective Token Controlcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025GraLoRA: Granular Low-Rank Adaptation for Parameter-Efficient Fine-Tuning
  2. 2026Beyond Low-Rank Parameterization: Narrowing the Gap Between LoRA and Full Fine-Tuning via Gradient Decomposition
  3. 2025Uni-LoRA: One Vector is All You Need
  4. 2025Less is More: Resource-Efficient Low-Rank Adaptation
  5. 2024Flat-LoRA: Low-Rank Adaptation over a Flat Loss Landscape
  6. 2026SOS-LoRA: Static Orthogonal-Subspace Low-Rank Adaptation with Fixed Multi-Scale Scaling
  7. 2025DropLoRA: Sparse Low-Rank Adaptation for Parameter-Efficient Fine-Tuning
  8. 2025LoRALib: A Standardized Benchmark for Evaluating LoRA-MoE Methods