mlsanityData to 14 Oct 2026

Per-Matrix Optimality Is Not Enough: Three-Level Optimization for Low-Rank LLM Compression

arXiv 2609.15838 ↗First posted 14 Sep 2026cs.LG cs.AIcited by 0

Not on the map. Why?

Abstract (arXiv)

Per-matrix singular value decomposition (SVD) truncation is Eckart-Young optimal in the whitened Frobenius norm, but errors from independently compressed matrices compound through the block's nonlinear forward pass. Inspired in part by hierarchical variational optimization in quantum many-body methods, we introduce a three-level chain that widens optimization scope from individual matrices to Transformer blocks to the full model: whitened SVD~(L1), block-level joint optimization~(L2), and end-to-end language-modeling loss refinement~(L3), all from 256 calibration sequences, with no instruction or recovery data. On LLaMA-7B at 60% compression, the chain reduces WikiText-2 perplexity from 42.1 to 19.1 to 11.4. The block-level stage acts as a regularizer: skipping it worsens Penn Treebank (PTB) perplexity by 24 points, a gap that additional end-to-end training did not close in our experiments. Perplexity gains hold across 20-80% compression, five architectures up to 13B parameters, and both in-distribution and out-of-distribution benchmarks, though the cross-architecture rows use architecture-specific configurations and the ratio sweep was not run under one common protocol. With more calibration data, skipping the block-level stage becomes competitive, revealing an offline compute--data trade-off. We therefore claim improvements only in perplexity and compression fidelity; downstream accuracy remains well below the dense model.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 18 of its 24 references that are in the map. First posted on arXiv 14 Sep 2026.

Builds on

18 of 24 references.

References · 18

  1. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  2. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
  3. 2015Distilling the Knowledge in a Neural Networkcited by 10,324
  4. 2016Pointer Sentinel Mixture Modelscited by 3,096
  5. 2022GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformerscited by 1,919
  6. 2023AWQ: Activation-aware Weight Quantization for LLM Compression and Accelerationcited by 1,298
  7. 2023SparseGPT: Massive Language Models Can Be Accurately Pruned in One-Shotcited by 1,121
  8. 2014Exploiting Linear Structure Within Convolutional Networks for Efficient Evaluationcited by 306
  9. 2014Speeding up Convolutional Neural Networks with Low Rank Expansionscited by 284
  10. 2024SVD-LLM: Truncation-aware Singular Value Decomposition for Large Language Model Compressioncited by 243
  11. 2026SAES-SVD: Self-Adaptive Suppression of Accumulated and Local Errors for SVD-based LLM Compressioncited by 12
  12. 2025DipSVD: Dual-importance Protected SVD for Efficient LLM Compressioncited by 8
6 more
  1. 2026Zero Sum SVD: Balancing Loss Sensitivity for Low Rank LLM Compressioncited by 8
  2. 2025ERC-SVD: Error-Controlled SVD for Large Language Model Compressioncited by 7
  3. 2026AA-SVD : Anchored and Adaptive SVD for Large Language Model Compressioncited by 7
  4. 2024Lillama: Large Language Models Compression via Low-Rank Feature Distillationcited by 4
  5. 2026Swift-SVD: Theoretical Optimality Meets Practical Efficiency in Low-Rank LLM Compressioncited by 2
  6. 2026IRDS: Interpretable RLVR Data Selection via Verifier-Coupled Sparse Autoencoder Coveragecited by 1

Built on by

Not cited yet.