mlsanityData to 7 Oct 2026

Parameter-Efficient Orthogonal Finetuning via Butterfly Factorization

arXiv 2311.06243 ↗First posted 10 Nov 2023cs.LG cs.AI cs.CLcited by 112

Not on the map. Why?

Abstract (arXiv)

Large foundation models are becoming ubiquitous, but training them from scratch is prohibitively expensive. Thus, efficiently adapting these powerful models to downstream tasks is increasingly important. In this paper, we study a principled finetuning paradigm -- Orthogonal Finetuning (OFT) -- for downstream task adaptation. Despite demonstrating good generalizability, OFT still uses a fairly large number of trainable parameters due to the high dimensionality of orthogonal matrices. To address this, we start by examining OFT from an information transmission perspective, and then identify a few key desiderata that enable better parameter-efficiency. Inspired by how the Cooley-Tukey fast Fourier transform algorithm enables efficient information transmission, we propose an efficient orthogonal parameterization using butterfly structures. We apply this parameterization to OFT, creating a novel parameter-efficient finetuning method, called Orthogonal Butterfly (BOFT). By subsuming OFT as a special case, BOFT introduces a generalized orthogonal finetuning framework. Finally, we conduct an extensive empirical study of adapting large vision transformers, large language models, and text-to-image diffusion models to various downstream tasks in vision and language.

Key facts. Cited by 112 papers in the map (data through 7 Oct 2026). Builds on 73 of its 104 references that are in the map. First posted on arXiv 10 Nov 2023.

Builds on

73 of 104 references.

References · 73

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  3. 2020Language Models are Few-Shot Learnerscited by 32,199
  4. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  5. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  6. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  7. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  8. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  9. 2020Denoising Diffusion Implicit Modelscited by 9,655
  10. 2023Segment Anythingcited by 8,158
  11. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  12. 2020Measuring Massive Multitask Language Understandingcited by 7,768
38 more
  1. 2023DINOv2: Learning Robust Visual Features without Supervisioncited by 7,764
  2. 2023Adding Conditional Control to Text-to-Image Diffusion Modelscited by 5,466
  3. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,854
  4. 2019Parameter-Efficient Transfer Learning for NLPcited by 4,195
  5. 2021Prefix-Tuning: Optimizing Continuous Prompts for Generationcited by 3,878
  6. 2021The Power of Scale for Parameter-Efficient Prompt Tuningcited by 3,769
  7. 2023QLoRA: Efficient Finetuning of Quantized LLMscited by 3,355
  8. 2022DreamBooth: Fine Tuning Text-to-Image Diffusion Models for Subject-Driven Generationcited by 3,286
  9. 2022An Image is Worth One Word: Personalizing Text-to-Image Generation using Textual Inversioncited by 2,231
  10. 2022Visual Prompt Tuningcited by 1,674
  11. 2018Spectral Normalization for Generative Adversarial Networkscited by 1,618
  12. 2023T2I-Adapter: Learning Adapters to Dig out More Controllable Ability for Text-to-Image Diffusion Modelscited by 1,339
  13. 2021BitFit: Simple Parameter-efficient Fine-tuning for Transformer-based Masked Language-modelscited by 1,234
  14. 2021DeBERTaV3: Improving DeBERTa using ELECTRA-Style Pre-Training with Gradient-Disentangled Embedding Sharingcited by 1,219
  15. 2022Multi-Concept Customization of Text-to-Image Diffusioncited by 1,099
  16. 2016Semantic Understanding of Scenes through the ADE20K Datasetcited by 1,063
  17. 2022Few-Shot Parameter-Efficient Fine-Tuning is Better and Cheaper than In-Context Learningcited by 886
  18. 2021Towards a Unified View of Parameter-Efficient Transfer Learningcited by 863
  19. 2020AdapterFusion: Non-Destructive Task Composition for Transfer Learningcited by 780
  20. 2023MetaMath: Bootstrap Your Own Mathematical Questions for Large Language Modelscited by 724
  21. 2022AdaptFormer: Adapting Vision Transformers for Scalable Visual Recognitioncited by 714
  22. 2020Intrinsic Dimensionality Explains the Effectiveness of Language Model Fine-Tuningcited by 658
  23. 2017Learning multiple visual domains with residual adapterscited by 553
  24. 2021Compacter: Efficient Low-Rank Hypercomplex Adapter Layerscited by 458
  25. 2019A Large-scale Study of Representation Learning with the Visual Task Adaptation Benchmarkcited by 419
  26. 2023Segment Anything in High Qualitycited by 373
  27. 2020Parameter-Efficient Transfer Learning with Diff Pruningcited by 351
  28. 2017Implicit Regularization in Matrix Factorizationcited by 318
  29. 2022Scaling & Shifting Your Features: A New Baseline for Efficient Model Tuningcited by 292
  30. 2022CelebV-HQ: A Large-Scale Video Facial Attributes Datasetcited by 246
  31. 2021SPoT: Better Frozen Model Adaptation through Soft Prompt Transfercited by 235
  32. 2022DyLoRA: Parameter Efficient Tuning of Pre-trained Models using Dynamic Search-Free Low-Rank Adaptationcited by 229
  33. 2022LST: Ladder Side-Tuning for Parameter and Memory Efficient Transfer Learningcited by 228
  34. 2021Training Neural Networks with Fixed Sparse Maskscited by 212
  35. 2023Controlling Text-to-Image Diffusion by Orthogonal Finetuningcited by 207
  36. 2017Spectral Norm Regularization for Improving the Generalizability of Deep Learningcited by 178
  37. 2023Scaling Down to Scale Up: A Guide to Parameter-Efficient Fine-Tuningcited by 171
  38. 2021UniPELT: A Unified Framework for Parameter-Efficient Language Model Tuningcited by 162

50 most cited of 73 All 73 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 35; 2025: 50; 2026: 27.

112 papers.

Sort:

By area:

  1. 2024DoRA: Weight-Decomposed Low-Rank Adaptationcited by 740
  2. 2024SWIFT:A Scalable lightWeight Infrastructure for Fine-Tuningcited by 345
  3. 2024ReFT: Representation Finetuning for Language Modelscited by 186
  4. 2024Asymmetry in Low-Rank Adapters of Foundation Modelscited by 81
  5. 2024SVFT: Parameter-Efficient Fine-Tuning with Singular Vectorscited by 57
  6. 2024Riemannian Preconditioned LoRA for Fine-Tuning Foundation Modelscited by 51
  7. 2024Towards Robust and Parameter-Efficient Knowledge Unlearning for LLMscited by 36
  8. 2025FlyLoRA: Boosting Task Decoupling and Parameter Efficiency via Implicit Rank-Wise Mixture-of-Expertscited by 34
  9. 2024Can Large Language Models Understand Symbolic Graphics Programs?cited by 32
  10. 2024Efficient Diversity-Preserving Diffusion Alignment via Gradient-Informed GFlowNetscited by 30
  11. 2024VERA: Explainable Video Anomaly Detection via Verbalized Learning of Vision-Language Modelscited by 24
  12. 2024ChatGarment: Garment Estimation, Generation and Editing via Large Language Modelscited by 22
38 more
  1. 2024Spectral Adapter: Fine-Tuning in Spectral Spacecited by 21
  2. 2025Personalized Image Generation with Deep Generative Models: A Decade Surveycited by 20
  3. 2025PoLAR: Polar-Decomposed Low-Rank Adapter Representationcited by 19
  4. 2024PA-SAM: Prompt Adapter SAM for High-Quality Image Segmentationcited by 17
  5. 2024ExPLoRA: Parameter-Efficient Extended Pre-Training to Adapt Vision Transformers under Domain Shiftscited by 16
  6. 2024Bridging The Gap between Low-rank and Orthogonal Adaptation via Householder Reflection Adaptationcited by 15
  7. 2024PACE: Marrying generalization in PArameter-efficient fine-tuning with Consistency rEgularizationcited by 14
  8. 2025Value Gradient Guidance for Flow Matching Alignmentcited by 14
  9. 2024NoRA: Nested Low-Rank Adaptation for Efficient Fine-Tuning Large Modelscited by 13
  10. 2025LoRA meets Riemannion: Muon Optimizer for Parametrization-independent Low-Rank Adapterscited by 13
  11. 2024LoRETTA: Low-Rank Economic Tensor-Train Adaptation for Ultra-Low-Parameter Fine-Tuning of Large Language Modelscited by 12
  12. 2025Generating Symbolic World Models via Test-time Scaling of Large Language Modelscited by 12
  13. 2025Nabla-R2D3: Effective and Efficient 3D Diffusion Alignment with 2D Rewardscited by 11
  14. 2024Group and Shuffle: Efficient Structured Orthogonal Parametrizationcited by 10
  15. 20243-in-1: 2D Rotary Adaptation for Efficient Finetuning, Efficient Batching and Composabilitycited by 10
  16. 2024Parameter-Efficient Fine-Tuning of Large Language Models using Semantic Knowledge Tuningcited by 10
  17. 2024S2FT: Efficient, Scalable and Generalizable LLM Fine-tuning by Structured Sparsitycited by 10
  18. 2025DeLoRA: Decoupling Angles and Strength in Low-rank Adaptationcited by 10
  19. 2026Dense Supervision, Sparse Updates: On the Sparsity and Geometry of On-Policy Distillationcited by 10
  20. 2024IntLoRA: Integral Low-rank Adaptation of Quantized Diffusion Modelscited by 9
  21. 2025Quantum-PEFT: Ultra parameter-efficient fine-tuningcited by 9
  22. 2025RefLoRA: Refactored Low-Rank Adaptation for Efficient Fine-Tuning of Large Modelscited by 9
  23. 2025Sharp Generalization Bounds for Foundation Models with Asymmetric Randomized Low-Rank Adapterscited by 9
  24. 2025SparseLoRA: Accelerating LLM Fine-Tuning with Contextual Sparsitycited by 9
  25. 2026Learning Rate Matters: Vanilla LoRA May Suffice for LLM Fine-tuningcited by 9
  26. 2024Reflecting on the State of Rehearsal-free Continual Learning with Pretrained Modelscited by 8
  27. 2025An Adaptive Orthogonal Convolution Scheme for Efficient and Flexible CNN Architecturescited by 8
  28. 2025Reparameterized LLM Training via Orthogonal Equivalence Transformationcited by 8
  29. 2025ButterflyQuant: Ultra-low-bit LLM Quantization through Learnable Orthogonal Butterfly Transformscited by 8
  30. 2024Vocabulary-Defined Semantics: Latent Space Clustering for Improving In-Context Learningcited by 7
  31. 2025Exploring Sparsity for Parameter Efficient Fine Tuning Using Wavelets for Visioncited by 7
  32. 2024Spectrum-Aware Parameter Efficient Fine-Tuning for Diffusion Modelscited by 5
  33. 2024Orthogonal Finetuning for Direct Preference Optimizationcited by 5
  34. 2025Train Small, Infer Large: Memory-Efficient LoRA Training for Large Language Modelscited by 5
  35. 2025PiCa: Parameter-Efficient Fine-Tuning with Column Space Projectioncited by 5
  36. 2024Parameter-Efficient Fine-Tuning via Circular Convolutioncited by 4
  37. 2025QuIC: Quantum-Inspired Compound Adapters for Parameter Efficient Fine-Tuningcited by 4
  38. 2025Faster Parameter-Efficient Tuning with Token Redundancy Reductioncited by 4

50 most cited of 112 All 112 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024Enhancing Robustness of Vision-Language Models through Orthogonality Learning and Self-Regularization
  2. 2026FRAME: Learning the Adaptation Domain with a Mixture of Fractional-Fourier Experts
  3. 2026Are Parameter-Efficient Fine-tuning Methods Really Different?
  4. 2026CapVector: Learning Transferable Capability Vectors in Parametric Space for Vision-Language-Action Models
  5. 2025PEFT A2Z: Parameter-Efficient Fine-Tuning Survey for Large Language and Vision Models
  6. 2026FourierMoE: Fourier Mixture-of-Experts Adaptation of Large Language Models
  7. 2025Rethinking Fine-Tuning: Unlocking Hidden Capabilities in Vision-Language Models
  8. 2025Towards Higher Effective Rank in Parameter-efficient Fine-tuning using Khatri--Rao Product