# Spectral Adapter: Fine-Tuning in Spectral Space

arXiv: https://arxiv.org/abs/2405.13952 · HTML: https://mlsanity.com/paper/2405.13952/

## Key facts

- Cited by 21 papers in the map (data through 14 Oct 2026).
- Builds on 39 of its 57 references that are in the map.
- First posted on arXiv 22 May 2024.

## Builds on (most cited first, 30 of 39)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [LoRA: Low-Rank Adaptation of Large Language Models](https://mlsanity.com/paper/2106.09685/) (2021)
- [Training Verifiers to Solve Math Word Problems](https://mlsanity.com/paper/2110.14168/) (2021)
- [Parameter-Efficient Transfer Learning for NLP](https://mlsanity.com/paper/1902.00751/) (2019)
- [Prefix-Tuning: Optimizing Continuous Prompts for Generation](https://mlsanity.com/paper/2101.00190/) (2021)
- [The Power of Scale for Parameter-Efficient Prompt Tuning](https://mlsanity.com/paper/2104.08691/) (2021)
- [Efficient Streaming Language Models with Attention Sinks](https://mlsanity.com/paper/2309.17453/) (2023)
- [T2I-Adapter: Learning Adapters to Dig out More Controllable Ability for Text-to-Image Diffusion Models](https://mlsanity.com/paper/2302.08453/) (2023)
- [AdapterFusion: Non-Destructive Task Composition for Transfer Learning](https://mlsanity.com/paper/2005.00247/) (2020)
- [Intrinsic Dimensionality Explains the Effectiveness of Language Model Fine-Tuning](https://mlsanity.com/paper/2012.13255/) (2020)
- [LoRA+: Efficient Low Rank Adaptation of Large Models](https://mlsanity.com/paper/2402.12354/) (2024)
- [Speeding up Convolutional Neural Networks with Low Rank Expansions](https://mlsanity.com/paper/1405.3866/) (2014)
- [Parameter-efficient Multi-task Fine-tuning for Transformers via Shared Hypernetworks](https://mlsanity.com/paper/2106.04489/) (2021)
- [Mix-of-Show: Decentralized Low-Rank Adaptation for Multi-Concept Customization of Diffusion Models](https://mlsanity.com/paper/2305.18292/) (2023)
- [SPoT: Better Frozen Model Adaptation through Soft Prompt Transfer](https://mlsanity.com/paper/2110.07904/) (2021)
- [DyLoRA: Parameter Efficient Tuning of Pre-trained Models using Dynamic Search-Free Low-Rank Adaptation](https://mlsanity.com/paper/2210.07558/) (2022)
- [Parameter-Efficient Fine-Tuning Methods for Pretrained Language Models: A Critical Review and Assessment](https://mlsanity.com/paper/2312.12148/) (2023)
- [Predicting Parameters in Deep Learning](https://mlsanity.com/paper/1306.0543/) (2013)
- [AdapterDrop: On the Efficiency of Adapters in Transformers](https://mlsanity.com/paper/2010.11918/) (2020)
- [WARP: Word-level Adversarial ReProgramming](https://mlsanity.com/paper/2101.00121/) (2021)
- [Implicit Self-Regularization in Deep Neural Networks: Evidence from Random Matrix Theory and Implications for Learning](https://mlsanity.com/paper/1810.01075/) (2018)
- [Controlling Text-to-Image Diffusion by Orthogonal Finetuning](https://mlsanity.com/paper/2306.07280/) (2023)
- [Exploring Versatile Generative Language Model Via Parameter-Efficient Transfer Learning](https://mlsanity.com/paper/2004.03829/) (2020)
- [LoRA-FA: Efficient and Effective Low Rank Representation Fine-tuning](https://mlsanity.com/paper/2308.03303/) (2023)
- [Parameter-Efficient Orthogonal Finetuning via Butterfly Factorization](https://mlsanity.com/paper/2311.06243/) (2023)
- [ATTEMPT: Parameter-Efficient Multi-task Tuning via Attentional Mixtures of Soft Prompts](https://mlsanity.com/paper/2205.11961/) (2022)
- [One-for-All: Generalized LoRA for Parameter-Efficient Fine-tuning](https://mlsanity.com/paper/2306.07967/) (2023)
- [AdapterSoup: Weight Averaging to Improve Generalization of Pretrained Language Models](https://mlsanity.com/paper/2302.07027/) (2023)
- [Delta-LoRA: Fine-Tuning High-Rank Parameters with the Delta of Low-Rank Matrices](https://mlsanity.com/paper/2309.02411/) (2023)
- [Orthogonal Adaptation for Modular Customization of Diffusion Models](https://mlsanity.com/paper/2312.02432/) (2023)

## Built on by (most cited first, 21 of 21)

- [See Further for Parameter Efficient Fine-tuning by Standing on the Shoulders of Decomposition](https://mlsanity.com/paper/2407.05417/) (2024)
- [StelLA: Subspace Learning in Low-rank Adaptation using Stiefel Manifold](https://mlsanity.com/paper/2510.01938/) (2025)
- [KIND: Knowledge Integration and Diversion for Training Decomposable Models](https://mlsanity.com/paper/2408.07337/) (2024)
- [Exploring Sparsity for Parameter Efficient Fine Tuning Using Wavelets for Vision](https://mlsanity.com/paper/2505.12532/) (2025)
- [LoLDU: Low-Rank Adaptation via Lower-Diag-Upper Decomposition for Parameter-Efficient Fine-Tuning](https://mlsanity.com/paper/2410.13618/) (2024)
- [Spectrum-Aware Parameter Efficient Fine-Tuning for Diffusion Models](https://mlsanity.com/paper/2405.21050/) (2024)
- [FINE: Factorizing Knowledge for Initialization of Variable-sized Diffusion Models](https://mlsanity.com/paper/2409.19289/) (2024)
- [Generalized Tensor-based Parameter-Efficient Fine-Tuning via Lie Group Transformations](https://mlsanity.com/paper/2504.00851/) (2025)
- [CoLA: Collaborative Low-Rank Adaptation](https://mlsanity.com/paper/2505.15471/) (2025)
- [Task-Specific Directions: Definition, Exploration, and Utilization in Parameter Efficient Fine-Tuning](https://mlsanity.com/paper/2409.01035/) (2024)
- [Efficient Orthogonal Fine-Tuning with Principal Subspace Adaptation](https://mlsanity.com/paper/2505.11235/) (2025)
- [ISO: An RLVR-Native Optimization Stack](https://mlsanity.com/paper/2607.19331/) (2026)
- [Alfa: Attentive Low-Rank Filter Adaptation for Structure-Aware Cross-Domain Personalized Gaze Estimation](https://mlsanity.com/paper/2603.08445/) (2026)
- [Curvature-Guided LoRA: Matching Full Fine-Tuning in Function Space](https://mlsanity.com/paper/2603.29824/) (2026)
- [HiCoLoRA: Addressing Context-Prompt Misalignment via Hierarchical Collaborative LoRA for Zero-Shot DST](https://mlsanity.com/paper/2509.19742/) (2025)
- [LOFT: Low-Rank Orthogonal Fine-Tuning via Task-Aware Support Selection](https://mlsanity.com/paper/2605.11872/) (2026)
- [FuRA: Full-Rank Parameter-Efficient Fine-Tuning with Spectral Preconditioning](https://mlsanity.com/paper/2605.22869/) (2026)
- [CRMA: A Spectrally-Bounded Backbone for Modular Continual Fine-Tuning of LLMs](https://mlsanity.com/paper/2606.00382/) (2026)
- [Spectral Adapters for Segment Anything Model-based Segmentation of Colorectal Liver Metastases in Computed Tomography](https://mlsanity.com/paper/2609.11703/) (2026)
- [Parameter-Efficient Fine-Tuning of Foundation Models for Liver Tumor Segmentation in CT](https://mlsanity.com/paper/2609.14106/) (2026)
- [Reuse or Relearn? A Spectral View of Earth Observation Foundation Models](https://mlsanity.com/paper/2609.32756/) (2026)

## Abstract (arXiv)

Recent developments in Parameter-Efficient Fine-Tuning (PEFT) methods for pretrained deep neural networks have captured widespread interest. In this work, we study the enhancement of current PEFT methods by incorporating the spectral information of pretrained weight matrices into the fine-tuning procedure. We investigate two spectral adaptation mechanisms, namely additive tuning and orthogonal rotation of the top singular vectors, both are done via first carrying out Singular Value Decomposition (SVD) of pretrained weights and then fine-tuning the top spectral space. We provide a theoretical analysis of spectral fine-tuning and show that our approach improves the rank capacity of low-rank adapters given a fixed trainable parameter budget. We show through extensive experiments that the proposed fine-tuning model enables better parameter efficiency and tuning performance as well as benefits multi-adapter fusion.