Q3R: Quadratic Reweighted Rank Regularizer for Effective Low-Rank Training

Parameter-efficient training based on low-rank optimization has become a highly successful tool for fine-tuning large deep learning models. However, these methods often fail for low-rank pre-training, where simultaneously maintaining low-rank weight structure and optimizing the task objective remains challenging. We propose the textitQuadratic Reweighted Rank Regularizer (textttQ3R), which leads to a novel low-rank-inducing training strategy inspired by the Iteratively Reweighted Least Squares (IRLS) framework. textttQ3R is based on a quadratic regularizer term that majorizes a smoothed log-determinant rank surrogate. Unlike other low-rank training techniques, textttQ3R can train weight matrices to prescribed low target ranks while achieving predictive performance comparable to dense models, with small computational overhead and full compatibility with existing architectures. For example, we demonstrate a textttQ3R-regularized ViT-Tiny experiment where truncating the model to 60% and 80% of its parameters results in only minor absolute accuracy drops of 1.3% and 4%, respectively, on CIFAR-10. We confirm the efficacy of textttQ3R on Transformers across both vision and language tasks, including low-rank fine-tuning.