mlsanityData to 7 Oct 2026

Spectral Adapter: Fine-Tuning in Spectral Space

arXiv 2405.13952 ↗First posted 22 May 2024cs.LG cs.AIcited by 21

Not on the map. Why?

Abstract (arXiv)

Recent developments in Parameter-Efficient Fine-Tuning (PEFT) methods for pretrained deep neural networks have captured widespread interest. In this work, we study the enhancement of current PEFT methods by incorporating the spectral information of pretrained weight matrices into the fine-tuning procedure. We investigate two spectral adaptation mechanisms, namely additive tuning and orthogonal rotation of the top singular vectors, both are done via first carrying out Singular Value Decomposition (SVD) of pretrained weights and then fine-tuning the top spectral space. We provide a theoretical analysis of spectral fine-tuning and show that our approach improves the rank capacity of low-rank adapters given a fixed trainable parameter budget. We show through extensive experiments that the proposed fine-tuning model enables better parameter efficiency and tuning performance as well as benefits multi-adapter fusion.

Key facts. Cited by 21 papers in the map (data through 7 Oct 2026). Builds on 39 of its 57 references that are in the map. First posted on arXiv 22 May 2024.

Builds on

39 of 57 references.

References · 39

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  3. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  4. 2019Parameter-Efficient Transfer Learning for NLPcited by 4,195
  5. 2021Prefix-Tuning: Optimizing Continuous Prompts for Generationcited by 3,878
  6. 2021The Power of Scale for Parameter-Efficient Prompt Tuningcited by 3,769
  7. 2023Efficient Streaming Language Models with Attention Sinkscited by 2,115
  8. 2023T2I-Adapter: Learning Adapters to Dig out More Controllable Ability for Text-to-Image Diffusion Modelscited by 1,339
  9. 2020AdapterFusion: Non-Destructive Task Composition for Transfer Learningcited by 780
  10. 2020Intrinsic Dimensionality Explains the Effectiveness of Language Model Fine-Tuningcited by 658
  11. 2024LoRA+: Efficient Low Rank Adaptation of Large Modelscited by 365
  12. 2014Speeding up Convolutional Neural Networks with Low Rank Expansionscited by 283
27 more
  1. 2021Parameter-efficient Multi-task Fine-tuning for Transformers via Shared Hypernetworkscited by 278
  2. 2023Mix-of-Show: Decentralized Low-Rank Adaptation for Multi-Concept Customization of Diffusion Modelscited by 261
  3. 2021SPoT: Better Frozen Model Adaptation through Soft Prompt Transfercited by 235
  4. 2022DyLoRA: Parameter Efficient Tuning of Pre-trained Models using Dynamic Search-Free Low-Rank Adaptationcited by 229
  5. 2023Parameter-Efficient Fine-Tuning Methods for Pretrained Language Models: A Critical Review and Assessmentcited by 227
  6. 2013Predicting Parameters in Deep Learningcited by 223
  7. 2020AdapterDrop: On the Efficiency of Adapters in Transformerscited by 223
  8. 2021WARP: Word-level Adversarial ReProgrammingcited by 220
  9. 2018Implicit Self-Regularization in Deep Neural Networks: Evidence from Random Matrix Theory and Implications for Learningcited by 211
  10. 2023Controlling Text-to-Image Diffusion by Orthogonal Finetuningcited by 207
  11. 2020Exploring Versatile Generative Language Model Via Parameter-Efficient Transfer Learningcited by 139
  12. 2023LoRA-FA: Efficient and Effective Low Rank Representation Fine-tuningcited by 132
  13. 2023Parameter-Efficient Orthogonal Finetuning via Butterfly Factorizationcited by 112
  14. 2022ATTEMPT: Parameter-Efficient Multi-task Tuning via Attentional Mixtures of Soft Promptscited by 85
  15. 2023AdapterSoup: Weight Averaging to Improve Generalization of Pretrained Language Modelscited by 84
  16. 2023One-for-All: Generalized LoRA for Parameter-Efficient Fine-tuningcited by 84
  17. 2023Delta-LoRA: Fine-Tuning High-Rank Parameters with the Delta of Low-Rank Matricescited by 72
  18. 2023Orthogonal Adaptation for Modular Customization of Diffusion Modelscited by 70
  19. 2023Conditional Adapters: Parameter-efficient Transfer Learning with Fast Inferencecited by 69
  20. 2020Few-Shot Adaptation of Generative Adversarial Networkscited by 64
  21. 2023Parameter Efficient Multi-task Model Fusion with Partial Linearizationcited by 64
  22. 2024Riemannian Preconditioned LoRA for Fine-Tuning Foundation Modelscited by 51
  23. 2023IncreLoRA: Incremental Parameter Allocation Method for Parameter-Efficient Fine-tuningcited by 45
  24. 2021Counter-Interference Adapter for Multilingual Machine Translationcited by 39
  25. 2023MerA: Merging Pretrained Adapters For Few-Shot Learningcited by 13
  26. 2022Tiny-Attention Adapter: Contexts Are More Important Than the Number of Parameterscited by 12
  27. 2022Empowering parameter-efficient transfer learning by recognizing the kernel structure in self-attentioncited by 6

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 6; 2025: 6; 2026: 9.

21 papers.

Sort:

By area:

  1. 2024See Further for Parameter Efficient Fine-tuning by Standing on the Shoulders of Decompositioncited by 13
  2. 2025StelLA: Subspace Learning in Low-rank Adaptation using Stiefel Manifoldcited by 13
  3. 2024KIND: Knowledge Integration and Diversion for Training Decomposable Modelscited by 10
  4. 2025Exploring Sparsity for Parameter Efficient Fine Tuning Using Wavelets for Visioncited by 7
  5. 2024LoLDU: Low-Rank Adaptation via Lower-Diag-Upper Decomposition for Parameter-Efficient Fine-Tuningcited by 6
  6. 2024Spectrum-Aware Parameter Efficient Fine-Tuning for Diffusion Modelscited by 5
  7. 2024FINE: Factorizing Knowledge for Initialization of Variable-sized Diffusion Modelscited by 5
  8. 2025CoLA: Collaborative Low-Rank Adaptationcited by 5
  9. 2024Task-Specific Directions: Definition, Exploration, and Utilization in Parameter Efficient Fine-Tuningcited by 4
  10. 2025Generalized Tensor-based Parameter-Efficient Fine-Tuning via Lie Group Transformationscited by 4
  11. 2025Efficient Orthogonal Fine-Tuning with Principal Subspace Adaptationcited by 4
  12. 2026Alfa: Attentive Low-Rank Filter Adaptation for Structure-Aware Cross-Domain Personalized Gaze Estimationcited by 1
9 more
  1. 2026Curvature-Guided LoRA: Matching Full Fine-Tuning in Function Spacecited by 1
  2. 2025HiCoLoRA: Addressing Context-Prompt Misalignment via Hierarchical Collaborative LoRA for Zero-Shot DSTcited by 0
  3. 2026LOFT: Low-Rank Orthogonal Fine-Tuning via Task-Aware Support Selectioncited by 0
  4. 2026FuRA: Full-Rank Parameter-Efficient Fine-Tuning with Spectral Preconditioningcited by 0
  5. 2026CRMA: A Spectrally-Bounded Backbone for Modular Continual Fine-Tuning of LLMscited by 0
  6. 2026ISO: An RLVR-Native Optimization Stackcited by 0
  7. 2026Spectral Adapters for Segment Anything Model-based Segmentation of Colorectal Liver Metastases in Computed Tomographycited by 0
  8. 2026Parameter-Efficient Fine-Tuning of Foundation Models for Liver Tumor Segmentation in CTcited by 0
  9. 2026Reuse or Relearn? A Spectral View of Earth Observation Foundation Modelscited by 0