# Graph-Based Spectral Decomposition for Parameter Coordination in Language Model Fine-Tuning

arXiv: https://arxiv.org/abs/2504.19583 · HTML: https://mlsanity.com/paper/2504.19583/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Builds on 10 of its 24 references that are in the map.
- First posted on arXiv 28 Apr 2025.

## Builds on (most cited first, 10 of 10)

- [LoRA: Low-Rank Adaptation of Large Language Models](https://mlsanity.com/paper/2106.09685/) (2021)
- [Instruction Tuning for Large Language Models: A Survey](https://mlsanity.com/paper/2308.10792/) (2023)
- [LLM-Adapters: An Adapter Family for Parameter-Efficient Fine-Tuning of Large Language Models](https://mlsanity.com/paper/2304.01933/) (2023)
- [LongLoRA: Efficient Fine-tuning of Long-Context Large Language Models](https://mlsanity.com/paper/2309.12307/) (2023)
- [Full Parameter Fine-tuning for Large Language Models with Limited Resources](https://mlsanity.com/paper/2306.09782/) (2023)
- [Social Network User Profiling for Anomaly Detection Based on Graph Neural Networks](https://mlsanity.com/paper/2503.19380/) (2025)
- [Addressing Class Imbalance with Probabilistic Graphical Models and Variational Inference](https://mlsanity.com/paper/2504.05758/) (2025)
- [Contrastive and Variational Approaches in Self-Supervised Learning for Complex Data Mining](https://mlsanity.com/paper/2504.04032/) (2025)
- [Context-Aware Adaptive Sampling for Intelligent Data Acquisition Systems Using DQN](https://mlsanity.com/paper/2504.09344/) (2025)
- [Dynamic Adaptation of LoRA Fine-Tuning for Efficient and Task-Specific Optimization of Large Language Models](https://mlsanity.com/paper/2501.14859/) (2025)

## Built on by (most cited first, 5 of 5)

- [Structured Memory Mechanisms for Stable Context Representation in Large Language Models](https://mlsanity.com/paper/2505.22921/) (2025)
- [Joint Graph Convolution and Sequential Modeling for Scalable Network Traffic Estimation](https://mlsanity.com/paper/2505.07674/) (2025)
- [Deep Probabilistic Modeling of User Behavior for Anomaly Detection via Mixture Density Networks](https://mlsanity.com/paper/2505.08220/) (2025)
- [Collaborative Distillation Strategies for Parameter-Efficient Language Model Deployment](https://mlsanity.com/paper/2507.15198/) (2025)
- [Graph Neural Network-Based Collaborative Perception for Adaptive Scheduling in Distributed Systems](https://mlsanity.com/paper/2505.16248/) (2025)

## Abstract (arXiv)

This paper proposes a parameter collaborative optimization algorithm for large language models, enhanced with graph spectral analysis. The goal is to improve both fine-tuning efficiency and structural awareness during training. In the proposed method, the parameters of a pre-trained language model are treated as nodes in a graph. A weighted graph is constructed, and Laplacian spectral decomposition is applied to enable frequency-domain modeling and structural representation of the parameter space. Based on this structure, a joint loss function is designed. It combines the task loss with a spectral regularization term to facilitate collaborative updates among parameters. In addition, a spectral filtering mechanism is introduced during the optimization phase. This mechanism adjusts gradients in a structure-aware manner, enhancing the model's training stability and convergence behavior. The method is evaluated on multiple tasks, including traditional fine-tuning comparisons, few-shot generalization tests, and convergence speed analysis. In all settings, the proposed approach demonstrates superior performance. The experimental results confirm that the spectral collaborative optimization framework effectively reduces parameter perturbations and improves fine-tuning quality while preserving overall model performance. This work contributes significantly to the field of artificial intelligence by advancing parameter-efficient training methodologies for large-scale models, reinforcing the importance of structural signal processing in deep learning optimization, and offering a robust, generalizable framework for enhancing language model adaptability and performance.