mlsanityData to 7 Oct 2026

Parameter-Efficient Fine-Tuning with Discrete Fourier Transform

arXiv 2405.03003 ↗First posted 5 May 2024cs.LG cs.AI cs.CLcited by 67

Not on the map. Why?

Abstract (arXiv)

Low-rank adaptation~(LoRA) has recently gained much interest in fine-tuning foundation models. It effectively reduces the number of trainable parameters by incorporating low-rank matrices A and B to represent the weight change, i.e., Delta W=BA. Despite LoRA's progress, it faces storage challenges when handling extensive customization adaptations or larger base models. In this work, we aim to further compress trainable parameters by enjoying the powerful expressiveness of the Fourier transform. Specifically, we introduce FourierFT, which treats Delta W as a matrix in the spatial domain and learns only a small fraction of its spectral coefficients. With the trained spectral coefficients, we implement the inverse discrete Fourier transform to recover Delta W. Empirically, our FourierFT method shows comparable or better performance with fewer parameters than LoRA on various tasks, including natural language understanding, natural language generation, instruction tuning, and image classification. For example, when performing instruction tuning on the LLaMA2-7B model, FourierFT surpasses LoRA with only 0.064M trainable parameters, compared to LoRA's 33.5M. Our code is released at urlhttps://github.com/Chaos96/fourierft.

Key facts. Cited by 67 papers in the map (data through 7 Oct 2026). Builds on 49 of its 69 references that are in the map. First posted on arXiv 5 May 2024.

Builds on

49 of 69 references.

References · 49

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  3. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  4. 2022Training language models to follow instructions with human feedbackcited by 15,712
  5. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  6. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  7. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  8. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  9. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  10. 2023Visual Instruction Tuningcited by 7,792
  11. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,854
  12. 2023Improved Baselines with Visual Instruction Tuningcited by 4,362
37 more
  1. 2019Parameter-Efficient Transfer Learning for NLPcited by 4,195
  2. 2021Prefix-Tuning: Optimizing Continuous Prompts for Generationcited by 3,878
  3. 2021The Power of Scale for Parameter-Efficient Prompt Tuningcited by 3,769
  4. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,747
  5. 2023QLoRA: Efficient Finetuning of Quantized LLMscited by 3,355
  6. 2022DreamBooth: Fine Tuning Text-to-Image Diffusion Models for Subject-Driven Generationcited by 3,286
  7. 2022Self-Instruct: Aligning Language Models with Self-Generated Instructionscited by 2,674
  8. 2020DeBERTa: Decoding-enhanced BERT with Disentangled Attentioncited by 2,157
  9. 2013Describing Textures in the Wildcited by 2,110
  10. 2013Fine-Grained Visual Classification of Aircraftcited by 1,701
  11. 2017EuroSAT: A Novel Dataset and Deep Learning Benchmark for Land Use and Land Cover Classificationcited by 1,488
  12. 2020Making Pre-trained Language Models Better Few-shot Learnerscited by 1,255
  13. 2021BitFit: Simple Parameter-efficient Fine-tuning for Transformer-based Masked Language-modelscited by 1,234
  14. 2021Towards a Unified View of Parameter-Efficient Transfer Learningcited by 863
  15. 2017Remote Sensing Image Scene Classification: Benchmark and State of the Artcited by 791
  16. 2020AdapterFusion: Non-Destructive Task Composition for Transfer Learningcited by 780
  17. 2021Cross-Task Generalization via Natural Language Crowdsourcing Instructionscited by 696
  18. 2021FLAVA: A Foundational Language And Vision Alignment Modelcited by 646
  19. 2021ImageNet-21K Pretraining for the Massescited by 607
  20. 2017Learning multiple visual domains with residual adapterscited by 553
  21. 2021Solving Inverse Problems in Medical Imaging with Score-Based Generative Modelscited by 514
  22. 2020Pre-trained Models for Natural Language Processing: A Surveycited by 504
  23. 2017The E2E Dataset: New Challenges For End-to-End Generationcited by 252
  24. 2022DyLoRA: Parameter Efficient Tuning of Pre-trained Models using Dynamic Search-Free Low-Rank Adaptationcited by 229
  25. 2020AdapterDrop: On the Efficiency of Adapters in Transformerscited by 223
  26. 2020Learning in the Frequency Domaincited by 196
  27. 2022Rethinking Graph Neural Networks for Anomaly Detectioncited by 185
  28. 2020Exploring Versatile Generative Language Model Via Parameter-Efficient Transfer Learningcited by 139
  29. 2022Black-box Prompt Learning for Pre-trained Language Modelscited by 70
  30. 2018Deep Residual Learning in the JPEG Transform Domaincited by 44
  31. 2019Sampled Softmax with Random Fourier Featurescited by 33
  32. 2024GraphWiz: An Instruction-Following Language Model for Graph Problemscited by 32
  33. 2022Handling Missing Data via Max-Entropy Regularized Graph Autoencodercited by 12
  34. 2024Protein Multimer Structure Prediction via Prompt Learningcited by 11
  35. 2013Spectral Compressed Sensing via Structured Matrix Completioncited by 7
  36. 2020Large-Scale Discrete Fourier Transform on TPUscited by 6
  37. 2022Community Question Answering Entity Linking via Leveraging Auxiliary Datacited by 5

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 15; 2025: 32; 2026: 20.

67 papers.

Sort:

By area:

  1. 2024SWIFT:A Scalable lightWeight Infrastructure for Fine-Tuningcited by 345
  2. 2024A Survey on LoRA of Large Language Modelscited by 67
  3. 2024Parameter-Efficient Fine-Tuning in Spectral Domain for Point Cloud Learningcited by 43
  4. 2024Visual Fourier Prompt Tuningcited by 30
  5. 2025AuroRA: Breaking Low-Rank Bottleneck of LoRA with Nonlinear Mappingcited by 22
  6. 2025LoRI: Reducing Cross-Task Interference in Multi-Task Low-Rank Adaptationcited by 18
  7. 2024Bridging The Gap between Low-rank and Orthogonal Adaptation via Householder Reflection Adaptationcited by 15
  8. 2024FouRA: Fourier Low Rank Adaptationcited by 15
  9. 2024On the Crucial Role of Initialization for Matrix Factorizationcited by 15
  10. 2025Keeping Yourself is Important in Downstream Tuning Multimodal Large Language Modelcited by 15
  11. 2025Attack and defense techniques in large language models: A survey and new perspectivescited by 13
  12. 2024Low-Rank Interconnected Adaptation across Layerscited by 10
38 more
  1. 2024SVFit: Parameter-Efficient Fine-Tuning of Large Pre-Trained Models Using Singular Valuescited by 10
  2. 2025DeLoRA: Decoupling Angles and Strength in Low-rank Adaptationcited by 10
  3. 2025RefLoRA: Refactored Low-Rank Adaptation for Efficient Fine-Tuning of Large Modelscited by 9
  4. 2026Learning Rate Matters: Vanilla LoRA May Suffice for LLM Fine-tuningcited by 9
  5. 2024PEANuT: Parameter-Efficient Adaptation with Weight-aware Neural Tweakerscited by 8
  6. 2024KaSA: Knowledge-Aware Singular-Value Adaptation of Large Language Modelscited by 8
  7. 2025GraLoRA: Granular Low-Rank Adaptation for Parameter-Efficient Fine-Tuningcited by 8
  8. 2025Exploring Sparsity for Parameter Efficient Fine Tuning Using Wavelets for Visioncited by 7
  9. 2025Uni-LoRA: One Vector is All You Needcited by 7
  10. 2025The Primacy of Magnitude in Low-Rank Adaptationcited by 7
  11. 2024LoLDU: Low-Rank Adaptation via Lower-Diag-Upper Decomposition for Parameter-Efficient Fine-Tuningcited by 6
  12. 2025MoST: Efficient Monarch Sparse Tuning for 3D Representation Learningcited by 6
  13. 2025Come Together, But Not Right Now: A Progressive Strategy to Boost Low-Rank Adaptationcited by 5
  14. 2024Parameter-Efficient Fine-Tuning via Circular Convolutioncited by 4
  15. 2024BiDoRA: Bi-level Optimization-Based Weight-Decomposed Low-Rank Adaptationcited by 4
  16. 2025LoCA: Location-Aware Cosine Adaptation for Parameter-Efficient Fine-Tuningcited by 4
  17. 2025Efficient Orthogonal Fine-Tuning with Principal Subspace Adaptationcited by 4
  18. 2025Dynamic Context-oriented Decomposition for Task-aware Low-rank Adaptation with Less Forgetting and Faster Convergencecited by 4
  19. 2025Transformed Low-rank Adaptation via Tensor Decomposition and Its Applications to Text-to-image Modelscited by 3
  20. 2025Instruction-Guided Autoregressive Neural Network Parameter Generationcited by 3
  21. 2025Block Circulant Adapter for Large Language Modelscited by 3
  22. 2025Parameter-Efficient Fine-Tuning with Circulant and Diagonal Vectorscited by 2
  23. 2025Language Ranker: A Lightweight Ranking framework for LLM Decodingcited by 2
  24. 2025Memory-Integrated Reconfigurable Adapters: A Unified Framework for Settings with Multiple Taskscited by 2
  25. 2025Delta Sampling: Data-Free Knowledge Transfer Across Diffusion Modelscited by 2
  26. 2026Low-Rank Adaptation Redux for Large Modelscited by 2
  27. 2025Meta-LoRA: Meta-Learning LoRA Components for Domain-Aware ID Personalizationcited by 1
  28. 20254,500 Seconds: Small Data Training Approaches for Deep UAV Audio Classificationcited by 1
  29. 2025WaRA: Wavelet Low-Rank Adaptation for Medical Image Classificationcited by 1
  30. 2025QWHA: Quantization-Aware Walsh-Hadamard Adaptation for Parameter-Efficient Fine-Tuning on Large Language Modelscited by 1
  31. 2025ScaLoRA: Optimally Scaled Low-Rank Adaptation for Efficient High-Rank Fine-Tuningcited by 1
  32. 2025Kernelized Sparse Fine-Tuning with Bi-level Parameter Competition for Vision Modelscited by 1
  33. 2025Near-Zero-Overhead Freshness for Recommendation Systems via Inference-Side Model Updatescited by 1
  34. 2026Nonlinearity as Rank: Generative Low-Rank Adapter with Radial Basis Functionscited by 1
  35. 2026Zipper-LoRA: Dynamic Parameter Decoupling for Speech-LLM based Multilingual Speech Recognitioncited by 1
  36. 2026Live Interactive Training for Video Segmentationcited by 1
  37. 2026S2FT: Parameter-Efficient Fine-Tuning in Sparse Spectrum Domaincited by 1
  38. 2026GPart: End-to-End Isometric Fine-Tuning via Global Parameter Partitioningcited by 1

50 most cited of 67 All 67 (JSON)