# LoRA-SP: Streamlined Partial Parameter Adaptation for Resource-Efficient Fine-Tuning of Large Language Models

arXiv: https://arxiv.org/abs/2403.08822 · HTML: https://mlsanity.com/paper/2403.08822/

## Key facts

- Cited by 9 papers in the map (data through 14 Oct 2026).
- Builds on 8 of its 19 references that are in the map.
- First posted on arXiv 28 Feb 2024.

## Builds on (most cited first, 8 of 8)

- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [Parameter-Efficient Transfer Learning for NLP](https://mlsanity.com/paper/1902.00751/) (2019)
- [LoRA Fine-tuning Efficiently Undoes Safety Training in Llama 2-Chat 70B](https://mlsanity.com/paper/2310.20624/) (2023)
- [LoRA-FA: Efficient and Effective Low Rank Representation Fine-tuning](https://mlsanity.com/paper/2308.03303/) (2023)
- [LoRA ensembles for large language model fine-tuning](https://mlsanity.com/paper/2310.00035/) (2023)
- [Research on Older Adults' Interaction with E-Health Interface Based on Explainable Artificial Intelligence](https://mlsanity.com/paper/2402.07915/) (2024)

## Built on by (most cited first, 9 of 9)

- [A Survey on LoRA of Large Language Models](https://mlsanity.com/paper/2407.11046/) (2024)
- [Parameter-Efficient Fine-Tuning in Large Models: A Survey of Methodologies](https://mlsanity.com/paper/2410.19878/) (2024)
- [Alignment and Safety in Large Language Models: Safety Mechanisms, Training Paradigms, and Emerging Challenges](https://mlsanity.com/paper/2507.19672/) (2025)
- [Flexora: Flexible Low Rank Adaptation for Large Language Models](https://mlsanity.com/paper/2408.10774/) (2024)
- [Less but Better: Parameter-Efficient Fine-Tuning of Large Language Models for Personality Detection](https://mlsanity.com/paper/2504.05411/) (2025)
- [Tiny-Critic RAG: Empowering Agentic Fallback with Parameter-Efficient Small Language Models](https://mlsanity.com/paper/2603.00846/) (2026)
- [DynaRAG: Bridging Static and Dynamic Knowledge in Retrieval-Augmented Generation](https://mlsanity.com/paper/2603.18012/) (2026)
- [CA-LoRA: Concept-Aware LoRA for Domain-Aligned Segmentation Dataset Generation](https://mlsanity.com/paper/2503.22172/) (2025)
- [RDP LoRA: Geometry-Driven Identification for Parameter-Efficient Adaptation in Large Language Models](https://mlsanity.com/paper/2604.19321/) (2026)

## Abstract (arXiv)

In addressing the computational and memory demands of fine-tuning Large Language Models(LLMs), we propose LoRA-SP(Streamlined Partial Parameter Adaptation), a novel approach utilizing randomized half-selective parameter freezing within the Low-Rank Adaptation(LoRA)framework. This method efficiently balances pre-trained knowledge retention and adaptability for task-specific optimizations. Through a randomized mechanism, LoRA-SP determines which parameters to update or freeze, significantly reducing computational and memory requirements without compromising model performance. We evaluated LoRA-SP across several benchmark NLP tasks, demonstrating its ability to achieve competitive performance with substantially lower resource consumption compared to traditional full-parameter fine-tuning and other parameter-efficient techniques. LoRA-SP innovative approach not only facilitates the deployment of advanced NLP models in resource-limited settings but also opens new research avenues into effective and efficient model adaptation strategies.