# Parameter-Efficient Fine-Tuning with Discrete Fourier Transform

arXiv: https://arxiv.org/abs/2405.03003 · HTML: https://mlsanity.com/paper/2405.03003/

## Key facts

- Cited by 71 papers in the map (data through 14 Oct 2026).
- Builds on 49 of its 69 references that are in the map.
- First posted on arXiv 5 May 2024.

## Builds on (most cited first, 30 of 49)

- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [High-Resolution Image Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2112.10752/) (2021)
- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [LLaMA: Open and Efficient Foundation Language Models](https://mlsanity.com/paper/2302.13971/) (2023)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://mlsanity.com/paper/2307.09288/) (2023)
- [Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena](https://mlsanity.com/paper/2306.05685/) (2023)
- [Visual Instruction Tuning](https://mlsanity.com/paper/2304.08485/) (2023)
- [GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding](https://mlsanity.com/paper/1804.07461/) (2018)
- [Improved Baselines with Visual Instruction Tuning](https://mlsanity.com/paper/2310.03744/) (2023)
- [Parameter-Efficient Transfer Learning for NLP](https://mlsanity.com/paper/1902.00751/) (2019)
- [Prefix-Tuning: Optimizing Continuous Prompts for Generation](https://mlsanity.com/paper/2101.00190/) (2021)
- [The Power of Scale for Parameter-Efficient Prompt Tuning](https://mlsanity.com/paper/2104.08691/) (2021)
- [Finetuned Language Models Are Zero-Shot Learners](https://mlsanity.com/paper/2109.01652/) (2021)
- [QLoRA: Efficient Finetuning of Quantized LLMs](https://mlsanity.com/paper/2305.14314/) (2023)
- [DreamBooth: Fine Tuning Text-to-Image Diffusion Models for Subject-Driven Generation](https://mlsanity.com/paper/2208.12242/) (2022)
- [Self-Instruct: Aligning Language Models with Self-Generated Instructions](https://mlsanity.com/paper/2212.10560/) (2022)
- [DeBERTa: Decoding-enhanced BERT with Disentangled Attention](https://mlsanity.com/paper/2006.03654/) (2020)
- [Describing Textures in the Wild](https://mlsanity.com/paper/1311.3618/) (2013)
- [Fine-Grained Visual Classification of Aircraft](https://mlsanity.com/paper/1306.5151/) (2013)
- [EuroSAT: A Novel Dataset and Deep Learning Benchmark for Land Use and Land Cover Classification](https://mlsanity.com/paper/1709.00029/) (2017)
- [Making Pre-trained Language Models Better Few-shot Learners](https://mlsanity.com/paper/2012.15723/) (2020)
- [BitFit: Simple Parameter-efficient Fine-tuning for Transformer-based Masked Language-models](https://mlsanity.com/paper/2106.10199/) (2021)
- [Towards a Unified View of Parameter-Efficient Transfer Learning](https://mlsanity.com/paper/2110.04366/) (2021)
- [Remote Sensing Image Scene Classification: Benchmark and State of the Art](https://mlsanity.com/paper/1703.00121/) (2017)
- [AdapterFusion: Non-Destructive Task Composition for Transfer Learning](https://mlsanity.com/paper/2005.00247/) (2020)
- [Cross-Task Generalization via Natural Language Crowdsourcing Instructions](https://mlsanity.com/paper/2104.08773/) (2021)
- [FLAVA: A Foundational Language And Vision Alignment Model](https://mlsanity.com/paper/2112.04482/) (2021)

## Built on by (most cited first, 30 of 71)

- [SWIFT:A Scalable lightWeight Infrastructure for Fine-Tuning](https://mlsanity.com/paper/2408.05517/) (2024)
- [A Survey on LoRA of Large Language Models](https://mlsanity.com/paper/2407.11046/) (2024)
- [Parameter-Efficient Fine-Tuning in Spectral Domain for Point Cloud Learning](https://mlsanity.com/paper/2410.08114/) (2024)
- [Visual Fourier Prompt Tuning](https://mlsanity.com/paper/2411.01327/) (2024)
- [AuroRA: Breaking Low-Rank Bottleneck of LoRA with Nonlinear Mapping](https://mlsanity.com/paper/2505.18738/) (2025)
- [LoRI: Reducing Cross-Task Interference in Multi-Task Low-Rank Adaptation](https://mlsanity.com/paper/2504.07448/) (2025)
- [Keeping Yourself is Important in Downstream Tuning Multimodal Large Language Model](https://mlsanity.com/paper/2503.04543/) (2025)
- [Bridging The Gap between Low-rank and Orthogonal Adaptation via Householder Reflection Adaptation](https://mlsanity.com/paper/2405.17484/) (2024)
- [FouRA: Fourier Low Rank Adaptation](https://mlsanity.com/paper/2406.08798/) (2024)
- [On the Crucial Role of Initialization for Matrix Factorization](https://mlsanity.com/paper/2410.18965/) (2024)
- [Attack and defense techniques in large language models: A survey and new perspectives](https://mlsanity.com/paper/2505.00976/) (2025)
- [SVFit: Parameter-Efficient Fine-Tuning of Large Pre-Trained Models Using Singular Values](https://mlsanity.com/paper/2409.05926/) (2024)
- [Low-Rank Interconnected Adaptation across Layers](https://mlsanity.com/paper/2407.09946/) (2024)
- [DeLoRA: Decoupling Angles and Strength in Low-rank Adaptation](https://mlsanity.com/paper/2503.18225/) (2025)
- [GraLoRA: Granular Low-Rank Adaptation for Parameter-Efficient Fine-Tuning](https://mlsanity.com/paper/2505.20355/) (2025)
- [Learning Rate Matters: Vanilla LoRA May Suffice for LLM Fine-tuning](https://mlsanity.com/paper/2602.04998/) (2026)
- [RefLoRA: Refactored Low-Rank Adaptation for Efficient Fine-Tuning of Large Models](https://mlsanity.com/paper/2505.18877/) (2025)
- [PEANuT: Parameter-Efficient Adaptation with Weight-aware Neural Tweakers](https://mlsanity.com/paper/2410.01870/) (2024)
- [KaSA: Knowledge-Aware Singular-Value Adaptation of Large Language Models](https://mlsanity.com/paper/2412.06071/) (2024)
- [Exploring Sparsity for Parameter Efficient Fine Tuning Using Wavelets for Vision](https://mlsanity.com/paper/2505.12532/) (2025)
- [Uni-LoRA: One Vector is All You Need](https://mlsanity.com/paper/2506.00799/) (2025)
- [The Primacy of Magnitude in Low-Rank Adaptation](https://mlsanity.com/paper/2507.06558/) (2025)
- [LoLDU: Low-Rank Adaptation via Lower-Diag-Upper Decomposition for Parameter-Efficient Fine-Tuning](https://mlsanity.com/paper/2410.13618/) (2024)
- [MoST: Efficient Monarch Sparse Tuning for 3D Representation Learning](https://mlsanity.com/paper/2503.18368/) (2025)
- [Come Together, But Not Right Now: A Progressive Strategy to Boost Low-Rank Adaptation](https://mlsanity.com/paper/2506.05713/) (2025)
- [Parameter-Efficient Fine-Tuning via Circular Convolution](https://mlsanity.com/paper/2407.19342/) (2024)
- [BiDoRA: Bi-level Optimization-Based Weight-Decomposed Low-Rank Adaptation](https://mlsanity.com/paper/2410.09758/) (2024)
- [LoCA: Location-Aware Cosine Adaptation for Parameter-Efficient Fine-Tuning](https://mlsanity.com/paper/2502.06820/) (2025)
- [Efficient Orthogonal Fine-Tuning with Principal Subspace Adaptation](https://mlsanity.com/paper/2505.11235/) (2025)
- [Dynamic Context-oriented Decomposition for Task-aware Low-rank Adaptation with Less Forgetting and Faster Convergence](https://mlsanity.com/paper/2506.13187/) (2025)

## Abstract (arXiv)

Low-rank adaptation~(LoRA) has recently gained much interest in fine-tuning foundation models. It effectively reduces the number of trainable parameters by incorporating low-rank matrices A and B to represent the weight change, i.e., Delta W=BA. Despite LoRA's progress, it faces storage challenges when handling extensive customization adaptations or larger base models. In this work, we aim to further compress trainable parameters by enjoying the powerful expressiveness of the Fourier transform. Specifically, we introduce FourierFT, which treats Delta W as a matrix in the spatial domain and learns only a small fraction of its spectral coefficients. With the trained spectral coefficients, we implement the inverse discrete Fourier transform to recover Delta W. Empirically, our FourierFT method shows comparable or better performance with fewer parameters than LoRA on various tasks, including natural language understanding, natural language generation, instruction tuning, and image classification. For example, when performing instruction tuning on the LLaMA2-7B model, FourierFT surpasses LoRA with only 0.064M trainable parameters, compared to LoRA's 33.5M. Our code is released at urlhttps://github.com/Chaos96/fourierft.