mlsanityData to 14 Oct 2026

Orthogonal Finetuning Made Scalable

arXiv 2506.19847 ↗First posted 24 Jun 2025cs.LG cs.AI cs.CLcited by 7

Not on the map. Why?

Abstract (arXiv)

Orthogonal finetuning (OFT) offers highly parameter-efficient adaptation while preventing catastrophic forgetting, but its high runtime and memory demands limit practical deployment. We identify the core computational bottleneck in OFT as its weight-centric implementation, which relies on costly matrix-matrix multiplications with cubic complexity. To overcome this, we propose OFTv2, an input-centric reformulation that instead uses matrix-vector multiplications (i.e., matrix-free computation), reducing the computational cost to quadratic. We further introduce the Cayley-Neumann parameterization, an efficient orthogonal parameterization that approximates the matrix inversion in the Cayley transform via a truncated Neumann series. These modifications allow OFTv2 to achieve up to 10x faster training and 3x lower GPU memory usage without compromising performance. In addition, we extend OFTv2 to support finetuning quantized foundation models and show that it outperforms the popular QLoRA in training stability, efficiency, and memory usage.

Key facts. Cited by 7 papers in the map (data through 14 Oct 2026). Builds on 54 of its 62 references that are in the map. First posted on arXiv 24 Jun 2025.

Builds on

54 of 62 references.

References · 54

  1. 2014Adam: A Method for Stochastic Optimizationcited by 37,240
  2. 2019BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehensioncited by 5,893
  3. 2019Parameter-Efficient Transfer Learning for NLPcited by 4,236
  4. 2024Scaling Rectified Flow Transformers for High-Resolution Image Synthesiscited by 4,181
  5. 2021Prefix-Tuning: Optimizing Continuous Prompts for Generationcited by 3,915
  6. 2021The Power of Scale for Parameter-Efficient Prompt Tuningcited by 3,795
  7. 2023QLoRA: Efficient Finetuning of Quantized LLMscited by 3,418
  8. 2022DreamBooth: Fine Tuning Text-to-Image Diffusion Models for Subject-Driven Generationcited by 3,300
  9. 2016Pointer Sentinel Mixture Modelscited by 3,096
  10. 2022Solving Quantitative Reasoning Problems with Language Modelscited by 1,392
  11. 2018Don't Give Me the Details, Just the Summary! Topic-Aware Convolutional Neural Networks for Extreme Summarizationcited by 1,317
  12. 2023AWQ: Activation-aware Weight Quantization for LLM Compression and Accelerationcited by 1,298
38 more
  1. 2021BitFit: Simple Parameter-efficient Fine-tuning for Transformer-based Masked Language-modelscited by 1,245
  2. 2024OlympiadBench: A Challenging Benchmark for Promoting AGI with Olympiad-Level Bilingual Multimodal Scientific Problemscited by 1,136
  3. 2015Teaching Machines to Read and Comprehendcited by 1,049
  4. 2022Few-Shot Parameter-Efficient Fine-Tuning is Better and Cheaper than In-Context Learningcited by 896
  5. 2021Towards a Unified View of Parameter-Efficient Transfer Learningcited by 872
  6. 2022AdaptFormer: Adapting Vision Transformers for Scalable Visual Recognitioncited by 719
  7. 2020Intrinsic Dimensionality Explains the Effectiveness of Language Model Fine-Tuningcited by 671
  8. 2017Program Induction by Rationale Generation : Learning to Solve and Explain Algebraic Word Problemscited by 650
  9. 2024QuaRot: Outlier-Free 4-Bit Inference in Rotated LLMscited by 634
  10. 2021Compacter: Efficient Low-Rank Hypercomplex Adapter Layerscited by 460
  11. 2024QuIP#: Even Better LLM Quantization with Hadamard Incoherence and Lattice Codebookscited by 378
  12. 2020Parameter-Efficient Transfer Learning with Diff Pruningcited by 352
  13. 2022Scaling & Shifting Your Features: A New Baseline for Efficient Model Tuningcited by 294
  14. 2021SPoT: Better Frozen Model Adaptation through Soft Prompt Transfercited by 235
  15. 2022DyLoRA: Parameter Efficient Tuning of Pre-trained Models using Dynamic Search-Free Low-Rank Adaptationcited by 233
  16. 2022LST: Ladder Side-Tuning for Parameter and Memory Efficient Transfer Learningcited by 230
  17. 2021Training Neural Networks with Fixed Sparse Maskscited by 214
  18. 2023Controlling Text-to-Image Diffusion by Orthogonal Finetuningcited by 208
  19. 2021UniPELT: A Unified Framework for Parameter-Efficient Language Model Tuningcited by 162
  20. 2021Composable Sparse Fine-Tuning for Cross-Lingual Transfercited by 144
  21. 2023Memory-Efficient Fine-Tuning of Compressed Large Language Models via sub-4-bit Integer Quantizationcited by 100
  22. 2023CMATH: Can Your Language Model Pass Chinese Elementary School Math Test?cited by 90
  23. 2024MARIO: MAth Reasoning with code Interpreter Output -- A Reproducible Pipelinecited by 81
  24. 2018Learning towards Minimum Hyperspherical Energycited by 75
  25. 2021Cross-Attention is All You Need: Adapting Pretrained Transformers for Machine Translationcited by 74
  26. 2023Delta-LoRA: Fine-Tuning High-Rank Parameters with the Delta of Low-Rank Matricescited by 74
  27. 2023Towards Efficient Visual Adaption via Structural Re-parameterizationcited by 73
  28. 2023Parameter-Efficient Fine-Tuning Design Spacescited by 67
  29. 2024SVFT: Parameter-Efficient Fine-Tuning with Singular Vectorscited by 58
  30. 2024Mixture-of-Subspaces in Low-Rank Adaptationcited by 47
  31. 2021Learning with Hyperspherical Uniformitycited by 35
  32. 2019Regularizing Neural Networks via Minimizing Hyperspherical Energycited by 25
  33. 2024Parameter Efficient Quasi-Orthogonal Fine-Tuning via Givens Rotationcited by 21
  34. 2024Bridging The Gap between Low-rank and Orthogonal Adaptation via Householder Reflection Adaptationcited by 15
  35. 2025OMoE: Diversifying Mixture of Low-Rank Adaptation by Orthogonal Finetuningcited by 15
  36. 2023Generalizing and Decoupling Neural Collapse via Hyperspherical Uniformity Gapcited by 14
  37. 2024ETHER: Efficient Finetuning of Large-Scale Models with Hyperplane Reflectionscited by 12
  38. 2024Group and Shuffle: Efficient Structured Orthogonal Parametrizationcited by 11

50 most cited of 54 All 54 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 7.

7 papers.

  1. 2026POET-X: Memory-efficient LLM Training by Scaling Orthogonal Transformationcited by 5
  2. 2026Orthogonal Model Mergingcited by 4
  3. 2026Orthogonal Concept Erasure for Diffusion Modelscited by 2
  4. 2026CORA: Per-Slice Coherent Orthogonal Rotation for SVD-based Low-Rank Adaptationcited by 1
  5. 2026LOFT: Low-Rank Orthogonal Fine-Tuning via Task-Aware Support Selectioncited by 0
  6. 2026Normalized Low-Rank Adaptationcited by 0
  7. 2026Multimodal Foundation Models Adaptation based on Domain-Aware Relaxed Orthogonal Subspace for Remote Sensingcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2023Parameter-Efficient Orthogonal Finetuning via Butterfly Factorization
  2. 2023ModuLoRA: Finetuning 2-Bit LLMs on Consumer GPUs by Integrating with Modular Quantizers
  3. 2026AutoQRA: Joint Optimization of Mixed-Precision Quantization and Low-rank Adapters for Efficient LLM Fine-Tuning
  4. 2024LoQT: Low-Rank Adapters for Quantized Pretraining
  5. 2025FALQON: Accelerating LoRA Fine-tuning with Low-Bit Floating-Point Arithmetic
  6. 2022QFT: Post-training quantization via fast joint finetuning of all degrees of freedom
  7. 2024Efficient Arbitrary Precision Acceleration for Large Language Models on GPU Tensor Cores
  8. 2026AQLoRA: A Zero-Search Recipe for Fast Quantized LoRA Fine-Tuning