LORA-CRAFT: Cross-layer Rank Adaptation via Frozen Tucker Decomposition of Pre-trained Attention Weights

We introduce LoRA-CRAFT (Cross-layer Rank Adaptation via Frozen Tucker), abbreviated CRAFT throughout, an extremely parameter-efficient fine-tuning (PEFT) method that applies Tucker tensor decomposition to pre-trained attention weight matrices stacked across transformer layers and trains only small square adaptation matrices on the resulting frozen Tucker factors. Existing tensor-based PEFT methods decompose textitgradient updates: LoTR applies Tucker decomposition with shared factor matrices, while SuperLoRA groups and reshapes Delta W across layers before applying Tucker decomposition. Separately, methods such as PiSSA apply SVD to textitpre-trained weights but operate independently per layer. CRAFT bridges these two lines of work: it performs full Tucker decomposition via Higher-Order SVD (HOSVD) directly on textitpre-trained weights organized as cross-layer 3D tensors, freezes all resulting factors, and adapts the model through lightweight trainable transformations applied to each factor matrix. Experiments on the GLUE benchmark using RoBERTa-base and RoBERTa-large, as well as commonsense reasoning benchmarks using LLaMA2-7B and LLaMA3-8B, demonstrate that CRAFT achieves competitive performance with existing methods while requiring only revextremely low Tucker adaptation parameters. fixwOn LLaMA3-8B, CRAFT revexceeds the average accuracy of LoRA using hundreds of times fewer parametersfixw; on LLaMA2-7B the same holds at a 0.252M budget. Our results suggest that CRAFT's efficiency advantage grows with model scale, as the frozen Tucker factors better capture the richer cross-layer structure of larger pre-trained models.