mlsanityData to 7 Oct 2026

Freeze the Decoder, Heal the Encoder: Parameter-Efficient Adaptation for SVD-Based KV-Cache Compression

arXiv 2610.10552 ↗First posted 25 Sep 2026cs.LG cs.AIcited by 0

On the map: Efficient LLM inference and serving → KV cache optimization

Abstract (arXiv)

Comparing parameter-efficient fine-tuning recipes under a single, shared learning rate is a common but flawed practice: when the arms being compared have very different trainable-parameter counts, a shared rate can simultaneously depress the larger arms' means and inflate their variance, manufacturing a large, seemingly multi-seed-significant advantage for the smallest arm that is not a real effect. We document this confound in a concrete setting: post-hoc SVD-based KV-cache compression, where an already-pretrained model is converted to a low-rank (multi-head-latent-attention-style) cache by factorizing its key/value weights into a down-projection ("encoder") and an up-projection ("decoder"), after which a short fine-tune ("healing") recovers the accuracy lost to truncation. Under a shared learning rate, freezing the decoder and healing only the encoder looks like a clear win over healing the decoder or both factors; once every arm is given its own tuned learning rate, that apparent advantage disappears, and encoder-only healing instead reaches parity with the alternatives, at a real, measured saving of 3x fewer trainable parameters and 3x less optimizer-state memory. We verify this parity with per-arm learning-rate tuning and three seeds per configuration on a vision-language model (Qwen2.5-VL-3B-Instruct), at the one compression ratio this protocol covers, and replicate it on a text-only testbed across two backbones. Encoder-only healing is therefore a lower-memory drop-in recipe for retrofitting low-rank KV-cache compression at training time, and the shared-learning-rate pitfall we document and correct is a cautionary result for comparing any fine-tuning recipes whose arms differ in trainable-parameter count.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in KV cache optimization, one of 12 directions in Efficient LLM inference and serving. Builds on 19 of its 31 references that are in the map. First posted on arXiv 25 Sep 2026.

Builds on

19 of 31 references.

Closest ancestors · 9

Same direction, newest first

  1. 2026CARE: Covariance-Aware and Rank-Enhanced Decomposition for Enabling Multi-Head Latent Attention18 Mar
  2. 2026KV-CoRE: Benchmarking Data-Dependent Low-Rank Compressibility of KV-Caches in LLMs5 Feb
  3. 2026Don't be so Stief! Learning KV Cache low-rank approximation over the Stiefel manifold29 Jan
  4. 2025KQ-SVD: Compressing the KV Cache with Provable Guarantees on Attention Fidelity5 Dec
  5. 2025KV Cache Transform Coding for Compact Storage in LLM Inference3 Nov
  6. 2025ReCalKV: Low-Rank KV Cache Compression via Head Reordering and Offline Calibration30 May
  7. 2025Towards Economical Inference: Enabling DeepSeek's Multi-Head Latent Attention in Any Transformer-based LLMs20 Feb
  8. 2024LoRC: Low-Rank Compression for LLMs KV Cache with a Progressive Compression Strategy4 Oct
  9. 2024Eigen Attention: Attention in Low-Rank Space for KV Cache Compression10 Aug

Other references · 10

  1. 2023GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpointscited by 1,311
  2. 2024DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Modelcited by 1,093
  3. 2025Native Sparse Attention: Hardware-Aligned and Natively Trainable Sparse Attentioncited by 391
  4. 2024LoRA Learns Less and Forgets Lesscited by 313
  5. 2024SVD-LLM: Truncation-aware Singular Value Decomposition for Large Language Model Compressioncited by 231
  6. 2024Asymmetry in Low-Rank Adapters of Foundation Modelscited by 81
  7. 2024MLKV: Multi-Layer Key-Value Heads for Memory Efficient Transformer Decodingcited by 20
  8. 2024Align Attention Heads Before Merging Them: An Effective Way for Converting MHA to GQAcited by 6
  9. 2026Zero Sum SVD: Balancing Loss Sensitivity for Low Rank LLM Compressioncited by 6
  10. 2026VideoMLA: Low-Rank Latent KV Cache for Minute-Scale Autoregressive Video Diffusioncited by 3

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026AttSVD:Prompt-Adaptive Low-Rank KV Cache Compression via Attention-Guided SVD
  2. 2026A JoLT for the KV cache: Near-Lossless KV Cache Compression via Joint Rank-bit Allocation
  3. 2026Behavior-Preserving KV Cache Compression
  4. 2025SmallKV: Small Model Assisted Compensation of KV Cache Compression for Efficient LLM Inference
  5. 2025CommonKV: Compressing KV Cache with Cross-layer Parameter Sharing
  6. 2026EchoKV: Efficient KV Cache Compression via Similarity-Based Reconstruction
  7. 2026Quality Recovery for Quantized KV Caches via Low-Rank Attention Adaptation
  8. 2026KV-Kaizen: Learning Context-Adaptive Cache Compression Choices