# FlashSVD v1.5: Making Low-Rank Transformers Inference Actually Fast

arXiv: https://arxiv.org/abs/2605.08314 · HTML: https://mlsanity.com/paper/2605.08314/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Placed in KV cache optimization, one of 12 directions in Efficient LLM inference and serving.
- Builds on 19 of its 30 references that are in the map.
- First posted on arXiv 8 May 2026.

## Placed in

- Efficient LLM inference and serving / [KV cache optimization](https://mlsanity.com/inference/kv-cache-optimization/)

## Builds on (most cited first, 19 of 19)

- [SVD-LLM: Truncation-aware Singular Value Decomposition for Large Language Model Compression](https://mlsanity.com/paper/2403.07378/) (2024)
- [Language model compression with weighted low-rank factorization](https://mlsanity.com/paper/2207.00112/) (2022)
- [ASVD: Activation-aware Singular Value Decomposition for Compressing Large Language Models](https://mlsanity.com/paper/2312.05821/) (2023)
- [Eigen Attention: Attention in Low-Rank Space for KV Cache Compression](https://mlsanity.com/paper/2408.05646/) (2024)
- [SVD-LLM V2: Optimizing Singular Value Truncation for Large Language Model Compression](https://mlsanity.com/paper/2503.12340/) (2025)
- [Palu: Compressing KV-Cache with Low-Rank Projection](https://mlsanity.com/paper/2407.21118/) (2024)
- [MoDeGPT: Modular Decomposition for Large Language Model Compression](https://mlsanity.com/paper/2408.09632/) (2024)
- [Basis Sharing: Cross-Layer Parameter Sharing for Large Language Model Compression](https://mlsanity.com/paper/2410.03765/) (2024)
- [QSVD: Efficient Low-rank Approximation for Unified Query-Key-Value Weight Compression in Low-Precision Vision-Language Models](https://mlsanity.com/paper/2510.16292/) (2025)
- [SAES-SVD: Self-Adaptive Suppression of Accumulated and Local Errors for SVD-based LLM Compression](https://mlsanity.com/paper/2602.03051/) (2026)
- [SADA: Stability-guided Adaptive Diffusion Acceleration](https://mlsanity.com/paper/2507.17135/) (2025)
- [Generalized Fisher-Weighted SVD: Scalable Kronecker-Factored Fisher Approximation for Compressing Large Language Models](https://mlsanity.com/paper/2505.17974/) (2025)
- [Layer-wise dynamic rank for compressing large language models](https://mlsanity.com/paper/2509.25622/) (2025)
- [Zero Sum SVD: Balancing Loss Sensitivity for Low Rank LLM Compression](https://mlsanity.com/paper/2602.02848/) (2026)
- [AA-SVD : Anchored and Adaptive SVD for Large Language Model Compression](https://mlsanity.com/paper/2604.02119/) (2026)
- [ZEUS: Accelerating Diffusion Models with Only Second-Order Predictor](https://mlsanity.com/paper/2604.01552/) (2026)
- [Scalable Dual Coordinate Descent for Kernel Methods](https://mlsanity.com/paper/2406.18001/) (2024)
- [Enhanced Cyclic Coordinate Descent Methods for Elastic Net Penalized Linear Models](https://mlsanity.com/paper/2510.19999/) (2025)
- [Blind Restoration of High-Resolution Ultrasound Video](https://mlsanity.com/paper/2505.13915/) (2025)

## Built on by (most cited first, 3 of 3)

- [DecodeShare: Tracing the Shared Subspace of LLM Decode-Time Decisions](https://mlsanity.com/paper/2607.20469/) (2026)
- [Target-Aware Calibration Data Selection for Preserving Uncertainty in Quantized Language Models](https://mlsanity.com/paper/2608.21019/) (2026)
- [ELDOR: A Dataset and Benchmark for Illegal Gold Mining in the Amazon Rainforest](https://mlsanity.com/paper/2605.15397/) (2026)

## Abstract (arXiv)

SVD-based Low-rank compression reduces transformer parameters and nominal FLOPs, but these savings often translate poorly into real LLM serving speedups. We show that this gap is largely a runtime problem: factorized checkpoints fragment execution paths, and the resulting overhead differs substantially between prefill and autoregressive decode. We present FlashSVD v1.5, a unified inference runtime for serving SVD-compressed transformers. FlashSVD v1.5 maps diverse public SVD compression families to a common factorized representation and combines phase-specific kernels with dense-KV decode, packed MLP execution, and per-layer CUDA-graph replay to reorganize the low-rank serving path into a thin runtime. Across representative decoder-serving settings, FlashSVD v1.5 achieves up to 2.55x decode and 2.39x end-to-end speedup, and it attains 1.48x average decode and 1.44x average end-to-end speedup across multiple popular SVD compression families. These results suggest that practical low-rank acceleration requires runtime co-design, not compression algorithms alone. Our code is available at: https://github.com/Zishan-Shao/FlashSVD.