mlsanityData to 7 Oct 2026

FlashSVD v1.5: Making Low-Rank Transformers Inference Actually Fast

arXiv 2605.08314 ↗First posted 8 May 2026cs.LG cs.AI cs.PFcited by 2

On the map: Efficient LLM inference and serving → KV cache optimization

Abstract (arXiv)

SVD-based Low-rank compression reduces transformer parameters and nominal FLOPs, but these savings often translate poorly into real LLM serving speedups. We show that this gap is largely a runtime problem: factorized checkpoints fragment execution paths, and the resulting overhead differs substantially between prefill and autoregressive decode. We present FlashSVD v1.5, a unified inference runtime for serving SVD-compressed transformers. FlashSVD v1.5 maps diverse public SVD compression families to a common factorized representation and combines phase-specific kernels with dense-KV decode, packed MLP execution, and per-layer CUDA-graph replay to reorganize the low-rank serving path into a thin runtime. Across representative decoder-serving settings, FlashSVD v1.5 achieves up to 2.55x decode and 2.39x end-to-end speedup, and it attains 1.48x average decode and 1.44x average end-to-end speedup across multiple popular SVD compression families. These results suggest that practical low-rank acceleration requires runtime co-design, not compression algorithms alone. Our code is available at: https://github.com/Zishan-Shao/FlashSVD.

Key facts. Cited by 2 papers in the map (data through 7 Oct 2026). Placed in KV cache optimization, one of 12 directions in Efficient LLM inference and serving. Builds on 19 of its 30 references that are in the map. First posted on arXiv 8 May 2026.

Builds on

19 of 30 references.

Closest ancestors · 2

Same direction, newest first

  1. 2024Eigen Attention: Attention in Low-Rank Space for KV Cache Compression10 Aug
  2. 2024Palu: Compressing KV-Cache with Low-Rank Projection30 Jul

Other references · 17

  1. 2024SVD-LLM: Truncation-aware Singular Value Decomposition for Large Language Model Compressioncited by 231
  2. 2022Language model compression with weighted low-rank factorizationcited by 192
  3. 2023ASVD: Activation-aware Singular Value Decomposition for Compressing Large Language Modelscited by 156
  4. 2025SVD-LLM V2: Optimizing Singular Value Truncation for Large Language Model Compressioncited by 63
  5. 2024MoDeGPT: Modular Decomposition for Large Language Model Compressioncited by 45
  6. 2024Basis Sharing: Cross-Layer Parameter Sharing for Large Language Model Compressioncited by 34
  7. 2025QSVD: Efficient Low-rank Approximation for Unified Query-Key-Value Weight Compression in Low-Precision Vision-Language Modelscited by 15
  8. 2025SADA: Stability-guided Adaptive Diffusion Accelerationcited by 9
  9. 2026SAES-SVD: Self-Adaptive Suppression of Accumulated and Local Errors for SVD-based LLM Compressioncited by 9
  10. 2025Generalized Fisher-Weighted SVD: Scalable Kronecker-Factored Fisher Approximation for Compressing Large Language Modelscited by 7
  11. 2025Layer-wise dynamic rank for compressing large language modelscited by 6
  12. 2026Zero Sum SVD: Balancing Loss Sensitivity for Low Rank LLM Compressioncited by 6
5 more
  1. 2026AA-SVD : Anchored and Adaptive SVD for Large Language Model Compressioncited by 5
  2. 2024Scalable Dual Coordinate Descent for Kernel Methodscited by 3
  3. 2026ZEUS: Accelerating Diffusion Models with Only Second-Order Predictorcited by 3
  4. 2025Enhanced Cyclic Coordinate Descent Methods for Elastic Net Penalized Linear Modelscited by 2
  5. 2025Blind Restoration of High-Resolution Ultrasound Videocited by 1

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 2.

2 papers.

  1. 2026DecodeShare: Tracing the Shared Subspace of LLM Decode-Time Decisionscited by 1
  2. 2026Target-Aware Calibration Data Selection for Preserving Uncertainty in Quantized Language Modelscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026FlashLoop: Fast and Memory-Efficient Looped Transformers via Lazy Updates
  2. 2022DeepSpeed Inference: Enabling Efficient Inference of Transformer Models at Unprecedented Scale
  3. 2026Importance-Aware Low-Rank Distillation of Diffusion Transformers
  4. 2024MLKV: Multi-Layer Key-Value Heads for Memory Efficient Transformer Decoding
  5. 2026The Residual Stream Is All You Need: On the Redundancy of the KV Cache in Transformer Inference
  6. 2026OCTOPUS: Optimized KV Cache for Transformers via Octahedral Parametrization Under optimal Squared error quantization
  7. 2026Swift-SVD: Theoretical Optimality Meets Practical Efficiency in Low-Rank LLM Compression
  8. 2026ZipServ: Fast and Memory-Efficient LLM Inference with Hardware-Aware Lossless Compression