Block Circulant Adapter for Large Language Models
Not on the map. Why?
Abstract (arXiv)
Fine-tuning large language models (LLMs) is difficult due to their huge model size. Recent Fourier domain-based methods show potential for reducing fine-tuning costs. We propose a block circulant matrix-based fine-tuning method with a stable training heuristic to leverage the properties of circulant matrices and one-dimensional Fourier transforms to reduce storage and computation costs. Experiments show that our method uses 14× less number of parameters than VeRA, 16× smaller than LoRA and 32× less FLOPs than FourierFT, while maintaining close or better task performance. Our approach presents a promising way in frequency domain to fine-tune large models on downstream tasks.
Key facts. Cited by 3 papers in the map (data through 14 Oct 2026). Builds on 31 of its 43 references that are in the map. First posted on arXiv 1 May 2025.
Builds on
31 of 43 references.
References · 31
- 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
- 2020Language Models are Few-Shot Learnerscited by 32,419
- 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
- 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,305
- 2023GPT-4 Technical Reportcited by 18,396
- 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
- 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
- 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,372
- 2021Training Verifiers to Solve Math Word Problemscited by 10,109
- 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
- 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,899
- 2019Parameter-Efficient Transfer Learning for NLPcited by 4,236
19 more
- 2016SQuAD: 100,000+ Questions for Machine Comprehension of Textcited by 4,148
- 2023QLoRA: Efficient Finetuning of Quantized LLMscited by 3,418
- 2021Fantastically Ordered Prompts and Where to Find Them: Overcoming Few-Shot Prompt Order Sensitivitycited by 1,135
- 2017SemEval-2017 Task 1: Semantic Textual Similarity - Multilingual and Cross-lingual Focused Evaluationcited by 1,001
- 2018Neural Network Acceptability Judgmentscited by 958
- 2022LST: Ladder Side-Tuning for Parameter and Memory Efficient Transfer Learningcited by 230
- 2023Parameter-Efficient Fine-Tuning Methods for Pretrained Language Models: A Critical Review and Assessmentcited by 230
- 2023Multitask Prompt Tuning Enables Parameter-Efficient Transfer Learningcited by 115
- 2022Monarch: Expressive Structured Matrices for Efficient and Accurate Trainingcited by 106
- 2024Parameter-Efficient Fine-Tuning with Discrete Fourier Transformcited by 71
- 2017CirCNN: Accelerating and Compressing Deep Neural Networks Using Block-CirculantWeight Matricescited by 55
- 2023AutoPEFT: Automatic Configuration Search for Parameter-Efficient Fine-Tuningcited by 49
- 2024Pre-trained Large Language Models Use Fourier Features to Compute Additioncited by 39
- 2023Prompt Sapper: A LLM-Empowered Production Tool for Building AI Chainscited by 24
- 2015ACDC: A Structured Efficient Linear Layercited by 19
- 2018Learning Compressed Transforms with Low Displacement Rankcited by 19
- 2024LaMDA: Large Model Fine-Tuning via Spectrally Decomposed Low-Dimensional Adaptationcited by 16
- 2017Theoretical Properties for Neural Networks with Weight Matrices of Low Displacement Rankcited by 11
- 2019CircConv: A Structured Convolution with Low Complexitycited by 8
Built on by
Citing papers per year: 2025: 2; 2026: 1.
3 papers.