mlsanityData to 13 Oct 2026

LOFT: Low-Rank Orthogonal Fine-Tuning via Task-Aware Support Selection

arXiv 2605.11872 ↗First posted 12 May 2026cs.LG stat.MLcited by 0

Not on the map. Why?

Abstract (arXiv)

Orthogonal parameter-efficient fine-tuning (PEFT) adapts pretrained weights through structure-preserving multiplicative transformations, but existing methods often conflate two distinct design choices: the subspace in which adaptation occurs and the transformation applied within that subspace. This paper introduces LOFT, a low-rank orthogonal fine-tuning framework that explicitly separates these two components. By viewing orthogonal adaptation as a multiplicative subspace rotation, LOFT provides a unified formulation that recovers representative orthogonal PEFT methods, including coordinate-, butterfly-, Householder-, and principal-subspace-based variants. More importantly, this perspective exposes support selection as a central design axis rather than a byproduct of a particular parameterization. We develop a first-order analysis showing that useful adaptation supports should be informed by the downstream training signal, motivating practical task-aware support selection strategies. Across language understanding, visual transfer, mathematical reasoning, and multilingual out-of-distribution adaptation, LOFT recovers principal-subspace orthogonal adaptation while gradient-informed supports improve the efficiency-performance trade-off under matched parameter, memory, and compute budgets. These results suggest that principled support selection is an important direction for improving orthogonal PEFT.

Key facts. No citing papers in the map yet (data through 13 Oct 2026). Builds on 36 of its 36 references that are in the map. First posted on arXiv 12 May 2026.

Builds on

36 of 36 references.

References · 36

  1. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,043
  2. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,231
  3. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,880
  4. 2021Prefix-Tuning: Optimizing Continuous Prompts for Generationcited by 3,898
  5. 2021The Power of Scale for Parameter-Efficient Prompt Tuningcited by 3,785
  6. 2023QLoRA: Efficient Finetuning of Quantized LLMscited by 3,388
  7. 2018Neural Tangent Kernel: Convergence and Generalization in Neural Networkscited by 2,610
  8. 2021BitFit: Simple Parameter-efficient Fine-tuning for Transformer-based Masked Language-modelscited by 1,240
  9. 2021DeBERTaV3: Improving DeBERTa using ELECTRA-Style Pre-Training with Gradient-Disentangled Embedding Sharingcited by 1,239
  10. 2022Few-Shot Parameter-Efficient Fine-Tuning is Better and Cheaper than In-Context Learningcited by 891
  11. 2024DoRA: Weight-Decomposed Low-Rank Adaptationcited by 747
  12. 2020Intrinsic Dimensionality Explains the Effectiveness of Language Model Fine-Tuningcited by 665
24 more
  1. 2024Parameter-Efficient Fine-Tuning for Large Models: A Comprehensive Surveycited by 652
  2. 2021Compacter: Efficient Low-Rank Hypercomplex Adapter Layerscited by 459
  3. 2024GaLore: Memory-Efficient LLM Training by Gradient Low-Rank Projectioncited by 456
  4. 2019A Large-scale Study of Representation Learning with the Visual Task Adaptation Benchmarkcited by 420
  5. 2018Measuring the Intrinsic Dimension of Objective Landscapescited by 370
  6. 2023AdaLoRA: Adaptive Budget Allocation for Parameter-Efficient Fine-Tuningcited by 347
  7. 2023VeRA: Vector-based Random Matrix Adaptationcited by 297
  8. 2024PiSSA: Principal Singular Values and Singular Vectors Adaptation of Large Language Modelscited by 265
  9. 2022DyLoRA: Parameter Efficient Tuning of Pre-trained Models using Dynamic Search-Free Low-Rank Adaptationcited by 231
  10. 2023Controlling Text-to-Image Diffusion by Orthogonal Finetuningcited by 207
  11. 2022A Kernel-Based View of Language Model Fine-Tuningcited by 119
  12. 2023Parameter-Efficient Orthogonal Finetuning via Butterfly Factorizationcited by 113
  13. 2024LoRA-XS: Low-Rank Adaptation with Extremely Small Number of Parameterscited by 75
  14. 2024SVFT: Parameter-Efficient Fine-Tuning with Singular Vectorscited by 58
  15. 2024LoRA Training in the NTK Regime has No Spurious Local Minimacited by 48
  16. 2020Orthogonal Over-Parameterized Trainingcited by 44
  17. 2024Parameter Efficient Quasi-Orthogonal Fine-Tuning via Givens Rotationcited by 21
  18. 2024Spectral Adapter: Fine-Tuning in Spectral Spacecited by 21
  19. 2024Bridging The Gap between Low-rank and Orthogonal Adaptation via Householder Reflection Adaptationcited by 15
  20. 2025LoFT: Low-Rank Adaptation That Behaves Like Full Fine-Tuningcited by 13
  21. 2025Orthogonal Finetuning Made Scalablecited by 7
  22. 2025Efficient Orthogonal Fine-Tuning with Principal Subspace Adaptationcited by 4
  23. 2025Efficient Optimization with Orthogonality Constraint: a Randomized Riemannian Submanifold Methodcited by 3
  24. 2025HOFT: Householder Orthogonal Fine-tuningcited by 3

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026SOS-LoRA: Static Orthogonal-Subspace Low-Rank Adaptation with Fixed Multi-Scale Scaling
  2. 2026Are Parameter-Efficient Fine-tuning Methods Really Different?
  3. 2026Parameter-Efficient Fine-Tuning with Learnable Rank
  4. 2024Initialization using Update Approximation is a Silver Bullet for Extremely Efficient Low-Rank Fine-Tuning
  5. 2026Parameter Efficiency Is Not Memory Efficiency: Rethinking Fine-Tuning for On-Device LLM Adaptation
  6. 2026ID-LoRA: Efficient Low-Rank Adaptation Inspired by Matrix Interpolative Decomposition
  7. 2025DropLoRA: Sparse Low-Rank Adaptation for Parameter-Efficient Fine-Tuning
  8. 2025Towards Higher Effective Rank in Parameter-efficient Fine-tuning using Khatri--Rao Product