mlsanityData to 7 Oct 2026

QuantMLA: Function-Aligned Dual-Path Quantization for Low-Bit MLA KV Caching

arXiv 2609.36760 ↗First posted 29 Sep 2026cs.LG cs.CLcited by 0

Not on the map. Why?

Abstract (arXiv)

Multi-Head Latent Attention (MLA) enables expressive multi-head attention with compact caches for its content and decoupled RoPE paths, yet cache memory still scales linearly with context length and batch size. In this work, we establish a systematic model of MLA's dual-path quantization errors, characterizing their distinct effects on attention-output distortion and explaining the pronounced amplification of RoPE-path errors. Guided by this analysis, we introduce QuantMLA, a function-aligned framework for low-bit dual-path quantization. We derive path-specific transformation spaces that preserve full-precision computation while remaining fully fusible into model parameters offline, eliminating online transformation overhead. Within these spaces, QuantMLA learns path-specific transformations with function-aligned objectives: attention-output reconstruction captures the content path's coupled matching and aggregation errors, while positional QK reconstruction preserves the RoPE-induced component of the attention logits and admits a theoretical bound on output distortion. Across four MLA model families, QuantMLA enables, to our knowledge, the first reported joint INT4 caching of the content and RoPE caches with minimal accuracy degradation. Further compressing the content cache to INT2 while retaining the RoPE key cache at INT4 maintains competitive performance on challenging reasoning and code benchmarks. We develop a native low-bit MLA attention kernel that integrates unpacking and dequantization directly into attention computation. The physical cache layout provides 3.59x compression at 128K context, while a cache-pressure serving workload achieves 5.168x higher whole-job output throughput than BF16. The code will be released upon acceptance.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 43 of its 50 references that are in the map. First posted on arXiv 29 Sep 2026.

Builds on

43 of 50 references.

References · 43

  1. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  2. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  3. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  4. 2020Measuring Massive Multitask Language Understandingcited by 7,768
  5. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,767
  6. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,203
  7. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,710
  8. 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,581
  9. 2016SQuAD: 100,000+ Questions for Machine Comprehension of Textcited by 4,115
  10. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,018
  11. 2016Pointer Sentinel Mixture Modelscited by 2,992
  12. 2019PIQA: Reasoning about Physical Commonsense in Natural Languagecited by 2,818
31 more
  1. 2023GPQA: A Graduate-Level Google-Proof Q&A Benchmarkcited by 2,748
  2. 2020Transformers are RNNs: Fast Autoregressive Transformers with Linear Attentioncited by 1,979
  3. 2024LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Codecited by 1,906
  4. 2019Root Mean Square Layer Normalizationcited by 1,425
  5. 2022SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Modelscited by 1,338
  6. 2024RULER: What's the Real Context Size of Your Long-Context Language Models?cited by 1,270
  7. 2024DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Modelcited by 1,093
  8. 2024KIVI: A Tuning-Free Asymmetric 2bit Quantization for KV Cachecited by 604
  9. 2024QuaRot: Outlier-Free 4-Bit Inference in Rotated LLMscited by 598
  10. 2024KVQuant: Towards 10 Million Context Length LLM Inference with KV Cache Quantizationcited by 571
  11. 2024SpinQuant: LLM quantization with learned rotationscited by 446
  12. 2023OmniQuant: Omnidirectionally Calibrated Quantization for Large Language Modelscited by 403
  13. 2025Native Sparse Attention: Hardware-Aligned and Natively Trainable Sparse Attentioncited by 391
  14. 2025Muon is Scalable for LLM Trainingcited by 381
  15. 2024Simple linear attention language models balance the recall-throughput tradeoffcited by 197
  16. 2025Kimi Linear: An Expressive, Efficient Attention Architecturecited by 169
  17. 2024DuQuant: Distributing Outliers via Dual Transformation Makes Stronger Quantized LLMscited by 165
  18. 2024FlatQuant: Flatness Matters for LLM Quantizationcited by 116
  19. 2025TurboQuant: Online Vector Quantization with Near-optimal Distortion Ratecited by 116
  20. 2023RPTQ: Reorder-based Post-training Quantization for Large Language Modelscited by 112
  21. 2025OstQuant: Refining Large Language Model Quantization with Orthogonal and Scaling Transformations for Better Distribution Fittingcited by 94
  22. 2025Quantization Hurts Reasoning? An Empirical Study on Quantized Reasoning Modelscited by 68
  23. 2025LongCat-Flash Technical Reportcited by 54
  24. 2025RotateKV: Accurate and Robust 2-Bit KV Cache Quantization for LLMs via Outlier-Aware Adaptive Rotationscited by 42
  25. 2024SKVQ: Sliding-window Key and Value Cache Quantization for Large Language Modelscited by 41
  26. 2025A Systematic Analysis of Hybrid Linear Attentioncited by 38
  27. 2025PolarQuant: Quantizing KV Caches with Polar Transformationcited by 24
  28. 2025FPTQuant: Function-Preserving Transforms for LLM Quantizationcited by 16
  29. 2026OSCAR: Offline Spectral Covariance-Aware Rotation for 2-bit KV Cache Quantizationcited by 7
  30. 2026OScaR: The Occam's Razor for Extreme KV Cache Quantization in LLMs and Beyondcited by 5
  31. 2026SnapMLA: Efficient Long-Context MLA Decoding via Hardware-Aware FP8 Quantized Pipeliningcited by 3

Built on by

Not cited yet.