mlsanityData to 14 Oct 2026

HOFT: Householder Orthogonal Fine-tuning

arXiv 2505.16531 ↗First posted 22 May 2025cs.LGcited by 3

Not on the map. Why?

Abstract (arXiv)

Adaptation of foundation models using low-rank methods is a widespread approach. Another way to adapt these models is to employ orthogonal fine-tuning methods, which are less time and memory efficient despite their good generalization properties. In this work, we propose Householder Orthogonal Fine-tuning (HOFT), a novel orthogonal fine-tuning method that aims to alleviate time and space complexity. Moreover, some theoretical properties of the orthogonal fine-tuning paradigm are explored. From this exploration, Scaled Householder Orthogonal Fine-tuning (SHOFT) is proposed. Both HOFT and SHOFT are evaluated in downstream tasks, namely commonsense reasoning, machine translation, subject-driven generation and mathematical reasoning. Compared with state-of-the-art adaptation methods, HOFT and SHOFT show comparable or better results.

Key facts. Cited by 3 papers in the map (data through 14 Oct 2026). Builds on 42 of its 55 references that are in the map. First posted on arXiv 22 May 2025.

Builds on

42 of 55 references.

References · 42

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  2. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,863
  3. 2024The Llama 3 Herd of Modelscited by 14,715
  4. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  5. 2018The Unreasonable Effectiveness of Deep Features as a Perceptual Metriccited by 11,177
  6. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  7. 2021Emerging Properties in Self-Supervised Vision Transformerscited by 6,968
  8. 2024Qwen2.5 Technical Reportcited by 4,942
  9. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,886
  10. 2023SDXL: Improving Latent Diffusion Models for High-Resolution Image Synthesiscited by 4,356
  11. 2019Parameter-Efficient Transfer Learning for NLPcited by 4,236
  12. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,100
30 more
  1. 2023QLoRA: Efficient Finetuning of Quantized LLMscited by 3,418
  2. 2022DreamBooth: Fine Tuning Text-to-Image Diffusion Models for Subject-Driven Generationcited by 3,300
  3. 2019PIQA: Reasoning about Physical Commonsense in Natural Languagecited by 2,883
  4. 2019BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questionscited by 2,183
  5. 2018Can a Suit of Armor Conduct Electricity? A New Dataset for Open Book Question Answeringcited by 2,164
  6. 2018A Call for Clarity in Reporting BLEU Scorescited by 1,799
  7. 2022No Language Left Behind: Scaling Human-Centered Machine Translationcited by 1,312
  8. 2020COMET: A Neural Framework for MT Evaluationcited by 1,033
  9. 2021Towards a Unified View of Parameter-Efficient Transfer Learningcited by 872
  10. 2024DoRA: Weight-Decomposed Low-Rank Adaptationcited by 758
  11. 2023MetaMath: Bootstrap Your Own Mathematical Questions for Large Language Modelscited by 735
  12. 2024Phi-4 Technical Reportcited by 711
  13. 2020Intrinsic Dimensionality Explains the Effectiveness of Language Model Fine-Tuningcited by 671
  14. 2024PiSSA: Principal Singular Values and Singular Vectors Adaptation of Large Language Modelscited by 269
  15. 2022DyLoRA: Parameter Efficient Tuning of Pre-trained Models using Dynamic Search-Free Low-Rank Adaptationcited by 233
  16. 2023Controlling Text-to-Image Diffusion by Orthogonal Finetuningcited by 208
  17. 2023QA-LoRA: Quantization-Aware Low-Rank Adaptation of Large Language Modelscited by 155
  18. 2024Orca-Math: Unlocking the potential of SLMs in Grade School Mathcited by 151
  19. 2021FedPara: Low-Rank Hadamard Product for Communication-Efficient Federated Learningcited by 141
  20. 2023Parameter-Efficient Orthogonal Finetuning via Butterfly Factorizationcited by 114
  21. 2023Navigating Text-To-Image Customization: From LyCORIS Fine-Tuning to Model Evaluationcited by 109
  22. 2018Learning towards Minimum Hyperspherical Energycited by 75
  23. 2024VB-LoRA: Extreme Parameter Efficient Fine-Tuning with Vector Bankscited by 52
  24. 2019Regularizing Neural Networks via Minimizing Hyperspherical Energycited by 25
  25. 2024Bridging The Gap between Low-rank and Orthogonal Adaptation via Householder Reflection Adaptationcited by 15
  26. 2024ELRT: Efficient Low-Rank Training for Compact Convolutional Neural Networkscited by 11
  27. 2022O-ViT: Orthogonal Vision Transformercited by 10
  28. 2025An Adaptive Orthogonal Convolution Scheme for Efficient and Flexible CNN Architecturescited by 8
  29. 2024RotRNN: Modelling Long Sequences with Rotationscited by 6
  30. 2023Towards Better Orthogonality Regularization with Disentangled Norm in Training Deep CNNscited by 3

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 3.

3 papers.

  1. 2026Deep Delta Learningcited by 9
  2. 2026LOFT: Low-Rank Orthogonal Fine-Tuning via Task-Aware Support Selectioncited by 0
  3. 2026LoCO: Low-rank Compositional Rotation Fine-tuningcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025HyperAdapt: Simple High-Rank Adaptation
  2. 2025Towards Higher Effective Rank in Parameter-efficient Fine-tuning using Khatri--Rao Product
  3. 2025HEFT: A Coarse-to-Fine Hierarchy for Enhancing the Efficiency and Accuracy of Language Model Reasoning
  4. 2025DiaBlo: Diagonal Blocks Are Sufficient For Finetuning
  5. 2026SOS-LoRA: Static Orthogonal-Subspace Low-Rank Adaptation with Fixed Multi-Scale Scaling
  6. 2025Singular Value Decomposition on Kronecker Adaptation for Large Language Model
  7. 2025UFT: Unifying Supervised and Reinforcement Fine-Tuning
  8. 2026Rotation-Preserving Supervised Fine-Tuning