CORA: Per-Slice Coherent Orthogonal Rotation for SVD-based Low-Rank Adaptation

Parameter-Efficient Fine-Tuning (PEFT) commonly adapts pretrained weights through low-rank updates, and recent methods further exploit the singular value decomposition (SVD) of the base weight for initialization or subspace selection. However, these methods do not explicitly preserve the coupled geometry between the pretrained left and right singular bases. Motivated by recent minimum-perturbation theory, which shows that stable finetuning follows a coherent SVD rotation in which a single orthogonal Q acts on both the left singular basis U0 and the right singular basis V0, we prove a per-slice analogue: each row slice of W0 can be adapted by a shared orthogonal rotation Qi on its left basis Ui and right basis Vi together with a diagonal spectrum shift. We implement this form as CORA (Coherent Orthogonal Rotation Adaptation), which applies per-slice orthogonal rotations and a per-layer diagonal scale to the rank-r SVD truncation of W0. CORA uses tfrac12m(r-1) trainable parameters per linear layer, about 4× fewer than LoRA at the same rank. CORA outperforms LoRA, DoRA, PiSSA, and MiLoRA on commonsense reasoning and code generation while using about 8× fewer parameters.