# Structure-Learnable Adapter Fine-Tuning for Parameter-Efficient Large Language Models

arXiv: https://arxiv.org/abs/2509.03057 · HTML: https://mlsanity.com/paper/2509.03057/

## Key facts

- Cited by 9 papers in the map (data through 14 Oct 2026).
- Builds on 15 of its 26 references that are in the map.
- First posted on arXiv 3 Sep 2025.

## Builds on (most cited first, 15 of 15)

- [Prefix-Tuning: Optimizing Continuous Prompts for Generation](https://mlsanity.com/paper/2101.00190/) (2021)
- [AdapterFusion: Non-Destructive Task Composition for Transfer Learning](https://mlsanity.com/paper/2005.00247/) (2020)
- [PiSSA: Principal Singular Values and Singular Vectors Adaptation of Large Language Models](https://mlsanity.com/paper/2404.02948/) (2024)
- [LongLoRA: Efficient Fine-tuning of Long-Context Large Language Models](https://mlsanity.com/paper/2309.12307/) (2023)
- [Raise a Child in Large Language Model: Towards Effective and Generalizable Fine-tuning](https://mlsanity.com/paper/2109.05687/) (2021)
- [Full Parameter Fine-tuning for Large Language Models with Limited Resources](https://mlsanity.com/paper/2306.09782/) (2023)
- [A Survey on LoRA of Large Language Models](https://mlsanity.com/paper/2407.11046/) (2024)
- [Fine-Tuning Large Neural Language Models for Biomedical Natural Language Processing](https://mlsanity.com/paper/2112.07869/) (2021)
- [Improving Large Language Model Fine-tuning for Solving Math Problems](https://mlsanity.com/paper/2310.10047/) (2023)
- [Automating Research Synthesis with Domain-Specific Large Language Model Fine-Tuning](https://mlsanity.com/paper/2404.08680/) (2024)
- [Multi-Agent Reinforcement Learning for Adaptive Resource Orchestration in Cloud-Native Clusters](https://mlsanity.com/paper/2508.10253/) (2025)
- [Collaborative Multi-Agent Reinforcement Learning Approach for Elastic Cloud Resource Scaling](https://mlsanity.com/paper/2507.00550/) (2025)
- [Structured Memory Mechanisms for Stable Context Representation in Large Language Models](https://mlsanity.com/paper/2505.22921/) (2025)
- [Clinical NLP with Attention-Based Deep Learning for Multi-Disease Prediction](https://mlsanity.com/paper/2507.01437/) (2025)
- [Collaborative Distillation Strategies for Parameter-Efficient Language Model Deployment](https://mlsanity.com/paper/2507.15198/) (2025)

## Built on by (most cited first, 9 of 9)

- [Trustworthy Summarization via Uncertainty Quantification and Risk Awareness in Large Language Models](https://mlsanity.com/paper/2510.01231/) (2025)
- [Modular Task Decomposition and Dynamic Collaboration in Multi-Agent Systems Driven by Large Language Models](https://mlsanity.com/paper/2511.01149/) (2025)
- [Deep Learning Approach for Clinical Risk Identification Using Transformer Modeling of Heterogeneous EHR Data](https://mlsanity.com/paper/2511.04158/) (2025)
- [Dynamic Topic Evolution with Temporal Decay and Attention in Large Language Models](https://mlsanity.com/paper/2510.10613/) (2025)
- [LLM-Centric RAG with Multi-Granular Indexing and Confidence Constraints](https://mlsanity.com/paper/2510.27054/) (2025)
- [Contrastive Knowledge Transfer and Robust Optimization for Secure Alignment of Large Language Models](https://mlsanity.com/paper/2510.27077/) (2025)
- [Structure-Aware Decoding Mechanisms for Complex Entity Extraction with Large-Scale Language Models](https://mlsanity.com/paper/2512.13980/) (2025)
- [Multi-Scale Feature Fusion and Graph Neural Network Integration for Text Classification with Large Language Models](https://mlsanity.com/paper/2511.05752/) (2025)
- [Machine Learning Approaches to Clinical Risk Prediction: Multi-Scale Temporal Alignment in Electronic Health Records](https://mlsanity.com/paper/2511.21561/) (2025)

## Abstract (arXiv)

This paper addresses the issues of parameter redundancy, rigid structure, and limited task adaptability in the fine-tuning of large language models. It proposes an adapter-based fine-tuning method built on a structure-learnable mechanism. By introducing differentiable gating functions and structural sparsity control variables, the method enables automatic optimization of adapter insertion points, activation paths, and module combinations. This allows the model to adjust its structure flexibly in multi-task settings to match different task characteristics. With the backbone parameters kept frozen, the method uses a structure search mechanism to guide the dynamic construction of task-specific efficient substructures during training. This significantly improves parameter utilization and representational capacity. In addition, the paper designs a set of sensitivity analysis experiments to systematically evaluate the effects of sparsity weight, noise injection ratio, and data perturbation on model performance. These experiments verify the stability and robustness of the proposed method across various multi-task natural language understanding tasks. The experimental results show that the proposed method outperforms mainstream parameter-efficient tuning techniques on multiple tasks. It achieves a better balance among accuracy, compression rate, and robustness to noise and perturbation.