mlsanityData to 14 Oct 2026

SAD-LoRA: Spectral Alignment for Low-Rank Knowledge Distillation

arXiv 2607.04306 ↗First posted 5 Jul 2026cs.LG cs.CVcited by 0

On the map: Knowledge distillation → General knowledge distillation methods

Abstract (arXiv)

Distilling a fine-tuned teacher into a LoRA-adapted student is a standard recipe for parameter-efficient compression, but output-level KD does not explicitly control which rank-r weight subspace the adapter occupies. We propose SAD-LoRA (Spectral Alignment Distillation), which selects this subspace from the data-weighted student-space reference update DWTSigx1/2 and maintains it during training via a differentiable principal-angle loss on colspan(B). We show that the data-weighted distillation error decomposes exactly into subspace misalignment, within-subspace coefficient mismatch, and irreducible rank residual; standard KD can affect the first term only indirectly through output gradients. On controlled synthetic problems with a flat teacher spectrum, SAD-LoRA reduces the subspace-misalignment term from 51% to nearly zero and lifts final subspace alignment from 0.49 to 1.00. On RoBERTa-large to RoBERTa-base distillation across six GLUE tasks, SAD-LoRA improves rank efficiency: at r=4, it matches or beats the strongest included spectral baseline on five of six tasks, and at r=8 it gives the best result on SST-2 and CoLA. Ablations identify subspace alignment as the load-bearing component, while coefficient matching is auxiliary.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in General knowledge distillation methods, one of 12 directions in Knowledge distillation. Builds on 19 of its 28 references that are in the map. First posted on arXiv 5 Jul 2026.

Builds on

19 of 28 references.

References · 19

  1. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
  2. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,372
  3. 2015Distilling the Knowledge in a Neural Networkcited by 10,324
  4. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,899
  5. 2019Parameter-Efficient Transfer Learning for NLPcited by 4,236
  6. 2021Prefix-Tuning: Optimizing Continuous Prompts for Generationcited by 3,915
  7. 2021The Power of Scale for Parameter-Efficient Prompt Tuningcited by 3,795
  8. 2023QLoRA: Efficient Finetuning of Quantized LLMscited by 3,418
  9. 2020Knowledge Distillation: A Surveycited by 1,603
  10. 2023AWQ: Activation-aware Weight Quantization for LLM Compression and Accelerationcited by 1,298
  11. 2021BitFit: Simple Parameter-efficient Fine-tuning for Transformer-based Masked Language-modelscited by 1,245
  12. 2023LoftQ: LoRA-Fine-Tuning-Aware Quantization for Large Language Modelscited by 196
7 more
  1. 2024LoRA vs Full Fine-tuning: An Illusion of Equivalencecited by 109
  2. 2024MiLoRA: Harnessing Minor Singular Components for Parameter-Efficient LLM Finetuningcited by 94
  3. 2023Tied-Lora: Enhancing parameter efficiency of LoRA with weight tyingcited by 65
  4. 2024VB-LoRA: Extreme Parameter Efficient Fine-Tuning with Vector Bankscited by 52
  5. 2023Seeking Neural Nuggets: Knowledge Transfer in Large Language Models from a Parametric Perspectivecited by 17
  6. 2024PC-LoRA: Low-Rank Adaptation for Progressive Model Compression with Knowledge Distillationcited by 9
  7. 2026Demystifying Low-Rank Knowledge Distillation in Large Language Models: Convergence, Generalization, and Information-Theoretic Guaranteescited by 3

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025Revisiting LoRA through the Lens of Parameter Redundancy: Spectral Encoding Helps
  2. 2026SOS-LoRA: Static Orthogonal-Subspace Low-Rank Adaptation with Fixed Multi-Scale Scaling
  3. 2024Mixture-of-Subspaces in Low-Rank Adaptation
  4. 2022Aligning Logits Generatively for Principled Black-Box Knowledge Distillation
  5. 2025LoRA-One: One-Step Full Gradient Could Suffice for Fine-Tuning Large Language Models, Provably and Efficiently
  6. 2024Initialization using Update Approximation is a Silver Bullet for Extremely Efficient Low-Rank Fine-Tuning
  7. 2024Flat-LoRA: Low-Rank Adaptation over a Flat Loss Landscape
  8. 2026Orthogonal Subspace Projection for Continual Machine Unlearning via SVD-Based LoRA