mlsanityData to 14 Oct 2026

StelLA: Subspace Learning in Low-rank Adaptation using Stiefel Manifold

arXiv 2510.01938 ↗First posted 2 Oct 2025cs.LGcited by 14

Not on the map. Why?

Abstract (arXiv)

Low-rank adaptation (LoRA) has been widely adopted as a parameter-efficient technique for fine-tuning large-scale pre-trained models. However, it still lags behind full fine-tuning in performance, partly due to its insufficient exploitation of the geometric structure underlying low-rank manifolds. In this paper, we propose a geometry-aware extension of LoRA that uses a three-factor decomposition U!SV^top. Analogous to the structure of singular value decomposition (SVD), it separates the adapter's input and output subspaces, V and U, from the scaling factor S. Our method constrains U and V to lie on the Stiefel manifold, ensuring their orthonormality throughout the training. To optimize on the Stiefel manifold, we employ a flexible and modular geometric optimization design that converts any Euclidean optimizer to a Riemannian one. It enables efficient subspace learning while remaining compatible with existing fine-tuning pipelines. Empirical results across a wide range of downstream tasks, including commonsense reasoning, math and code generation, image classification, and image generation, demonstrate the superior performance of our approach against the recent state-of-the-art variants of LoRA. Code is available at https://github.com/SonyResearch/stella.

Key facts. Cited by 14 papers in the map (data through 14 Oct 2026). Builds on 47 of its 74 references that are in the map. First posted on arXiv 2 Oct 2025.

Builds on

47 of 74 references.

References · 47

  1. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  3. 2020Language Models are Few-Shot Learnerscited by 32,419
  4. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  5. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,863
  6. 2023GPT-4 Technical Reportcited by 18,396
  7. 2024The Llama 3 Herd of Modelscited by 14,715
  8. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
  9. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  10. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  11. 2021Evaluating Large Language Models Trained on Codecited by 8,320
  12. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,371
35 more
  1. 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generationcited by 4,322
  2. 2019Parameter-Efficient Transfer Learning for NLPcited by 4,236
  3. 2021Prefix-Tuning: Optimizing Continuous Prompts for Generationcited by 3,915
  4. 2021The Power of Scale for Parameter-Efficient Prompt Tuningcited by 3,795
  5. 2021Program Synthesis with Large Language Modelscited by 3,637
  6. 2023QLoRA: Efficient Finetuning of Quantized LLMscited by 3,418
  7. 2023BloombergGPT: A Large Language Model for Financecited by 939
  8. 2013Exact solutions to the nonlinear dynamics of learning in deep linear neural networkscited by 763
  9. 2024DoRA: Weight-Decomposed Low-Rank Adaptationcited by 758
  10. 2023MetaMath: Bootstrap Your Own Mathematical Questions for Large Language Modelscited by 735
  11. 2024Parameter-Efficient Fine-Tuning for Large Models: A Comprehensive Surveycited by 656
  12. 2024LoRA+: Efficient Low Rank Adaptation of Large Modelscited by 377
  13. 2024VILA-U: a Unified Foundation Model Integrating Visual Understanding and Generationcited by 295
  14. 2024PiSSA: Principal Singular Values and Singular Vectors Adaptation of Large Language Modelscited by 269
  15. 2024OpenCodeInterpreter: Integrating Code Generation with Execution and Refinementcited by 247
  16. 2023A Rank Stabilization Scaling Factor for Fine-Tuning with LoRAcited by 243
  17. 2023Controlling Text-to-Image Diffusion by Orthogonal Finetuningcited by 208
  18. 2023ReLoRA: High-Rank Training Through Low-Rank Updatescited by 204
  19. 2018Riemannian Adaptive Optimization Methodscited by 194
  20. 2024MiLoRA: Harnessing Minor Singular Components for Parameter-Efficient LLM Finetuningcited by 94
  21. 2024LoRA-XS: Low-Rank Adaptation with Extremely Small Number of Parameterscited by 76
  22. 2024Riemannian Preconditioned LoRA for Fine-Tuning Foundation Modelscited by 52
  23. 2024Mixture-of-Subspaces in Low-Rank Adaptationcited by 47
  24. 2024OLoRA: Orthonormal Low-Rank Adaptation of Large Language Modelscited by 39
  25. 2023Tensor Programs IVb: Adaptive Optimization in the Infinite-Width Limitcited by 38
  26. 2024Spectral Adapter: Fine-Tuning in Spectral Spacecited by 21
  27. 2025PoLAR: Polar-Decomposed Low-Rank Adapter Representationcited by 20
  28. 2024Evaluating Large Language Models in Analysing Classroom Dialoguecited by 19
  29. 2024LaMDA: Large Model Fine-Tuning via Spectrally Decomposed Low-Dimensional Adaptationcited by 16
  30. 2023Robust low-rank training via approximate orthonormal constraintscited by 13
  31. 2023Geometry-aware training of factorized layers in tensor Tucker formatcited by 12
  32. 2013R3MC: A Riemannian three-factor algorithm for low-rank matrix completioncited by 11
  33. 2025DeLoRA: Decoupling Angles and Strength in Low-rank Adaptationcited by 10
  34. 2024Brain-Inspired Online Adaptation for Remote Sensing with Spiking Neural Networkcited by 8
  35. 2025Dynamical Low-Rank Compression of Neural Networks with Robustness under Adversarial Attackscited by 6

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 14.

14 papers.

Sort:

By area:

  1. 2026Spectral Imbalance Causes Forgetting in Low-Rank Continual Adaptationcited by 5
  2. 2026Don't be so Stief! Learning KV Cache low-rank approximation over the Stiefel manifoldcited by 3
  3. 2026Low-Rank Adaptation Redux for Large Modelscited by 3
  4. 2026ISO: An RLVR-Native Optimization Stackcited by 2
  5. 2026SDS-LoRA: Overcoming Anisotropic Gradient Scaling in Low-Rank Adaptationcited by 1
  6. 2026Spectral Compact Training: Pre-Training Large Language Models via Permanent Truncated SVD and Stiefel QR Retractioncited by 0
  7. 2026Efficient Pre-Training of LLMs through Truncated SVD Representationscited by 0
  8. 2026When Top-1 Fails: Calibrating LoRA Monitors for Masked Diffusion LMscited by 0
  9. 2026Chebyshev Manifold Adaptationcited by 0
  10. 2026Multivariate Scientific Data Compression with Learned Cross-Variable Latent Decorrelation and Autoregressive Entropy Modelingcited by 0
  11. 2026Behaviorally Effective LoRA Writes Are Sparse and Structuredcited by 0
  12. 2026Geometry-Aware Bayesian Parameter-Efficient Fine-Tuning on the Stiefel Manifold via Stein Variational Gradient Descentcited by 0
2 more
  1. 2026Behavior Quotient Learning for Low-Rank Adaptation of LLM Agentscited by 0
  2. 2026Safe Context Switching for Agents in the Wild: Mitigating Subspace Interference via Orthogonal Adaptationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Rotated Manifold Optimization for Low-Rank Adaptation
  2. 2025A Stronger Mixture of Low-Rank Experts for Fine-Tuning Foundation Models
  3. 2026SOS-LoRA: Static Orthogonal-Subspace Low-Rank Adaptation with Fixed Multi-Scale Scaling
  4. 2025Mitigating Forgetting in Low Rank Adaptation
  5. 2024Flat-LoRA: Low-Rank Adaptation over a Flat Loss Landscape
  6. 2026GeoRA: Geometry-Aware Low-Rank Adaptation for RLVR
  7. 2025Revisiting LoRA through the Lens of Parameter Redundancy: Spectral Encoding Helps
  8. 2026Beyond Low-Rank Parameterization: Narrowing the Gap Between LoRA and Full Fine-Tuning via Gradient Decomposition