mlsanityData to 7 Oct 2026

Regularize or Localize: When Training-Time KV-Cache Geometry Pays Under Quantization

arXiv 2607.17019 ↗First posted 19 Jul 2026cs.LGcited by 0

On the map: Efficient LLM inference and serving → KV cache quantization

Abstract (arXiv)

We study whether sigreg -- LeJEPA's anti-collapse objective -- can reshape representations during standard autoregressive language-model pretraining, and when the resulting geometry helps kv-cache quantization. We train 110M-parameter models on 10B FineWeb tokens and report three findings. (1) At λ=0.01, sigreg reduces hidden-state pairwise-cosine anisotropy by 38% across three paired seeds. Perplexity increases by less than 0.35% in every pair, with no consistent zero-shot loss. (2) This change does not propagate from hidden states to the kv cache. Applying sigreg directly to K and V during continued training, however, reduces mean cache anisotropy by 94% across four checkpoints. A matched continuation without the kv term leaves cache geometry nearly unchanged, and the frozen-trunk retrofits we tested do not reproduce the effect. (3) Under untransformed symmetric group-free quantization, direct kv regularization is the only training condition that prefers per-channel scaling in all three seeds, and under that same 3-bit per-channel scheme the baseline incurs 4.3--7.9× the directly regularized model's dnll. Under the full simulated KIVI-style configuration (mixed arrangement, zero-points, grouped scales), however, all models reach near-parity, including when storage overhead is approximately matched. In this 110M regime, the training intervention helps when quantizer scales are coarse; the advantage vanishes under the tested combination of token-local grouping, mixed kv scaling, and zero-points. To our knowledge this is the first training-time distributional regularization of standard kv-cache geometry evaluated against post-hoc cache quantization.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in KV cache quantization, one of 12 directions in Efficient LLM inference and serving. Builds on 36 of its 40 references that are in the map. First posted on arXiv 19 Jul 2026.

Builds on

36 of 40 references.

Closest ancestors · 6

Same direction, newest first

  1. 2026KVarN: Variance-Normalized KV-Cache Quantization Mitigates Error Accumulation in Reasoning Tasks2 Jun
  2. 2025NSNQuant: A Double Normalization Approach for Calibration-Free Low-Bit Vector Quantization of KV Cache23 May
  3. 2025Quantize What Counts: More for Keys, Less for Values20 Feb
  4. 2024QuaRot: Outlier-Free 4-Bit Inference in Rotated LLMs30 Mar
  5. 2024KIVI: A Tuning-Free Asymmetric 2bit Quantization for KV Cache5 Feb
  6. 2024KVQuant: Towards 10 Million Context Length LLM Inference with KV Cache Quantization31 Jan

Other references · 30

  1. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  2. 2023Mistral 7Bcited by 3,065
  3. 2023Efficient Streaming Language Models with Attention Sinkscited by 2,115
  4. 2024The FineWeb Datasets: Decanting the Web for the Finest Text Data at Scalecited by 1,104
  5. 2019How Contextual are Contextualized Word Representations? Comparing the Geometry of BERT, ELMo, and GPT-2 Embeddingscited by 770
  6. 2023LLM-QAT: Data-Free Quantization Aware Training for Large Language Modelscited by 320
  7. 2025Gated Attention for Large Language Models: Non-linearity, Sparsity, and Attention-Sink-Freecited by 266
  8. 2019Representation Degeneration Problem in Training Natural Language Generation Modelscited by 250
  9. 2025LeJEPA: Provable and Scalable Self-Supervised Learning Without the Heuristicscited by 227
  10. 2022A Contrastive Framework for Neural Text Generationcited by 224
  11. 2017All-but-the-Top: Simple and Effective Postprocessing for Word Representationscited by 205
  12. 2021All Bark and No Bite: Rogue Dimensions in Transformer Language Models Obscure Representational Qualitycited by 150
18 more
  1. 2023Quantizable Transformers: Removing Outliers by Helping Attention Heads Do Nothingcited by 147
  2. 2024Dynamic Memory Compression: Retrofitting LLMs for Accelerated Inferencecited by 115
  3. 2024Anisotropy Is Inherent to Self-Attention in Transformerscited by 62
  4. 2025LLM-JEPA: Large Language Models Meet Joint Embedding Predictive Architecturescited by 35
  5. 2025Next-Latent Prediction Transformers Learn Compact World Modelscited by 29
  6. 2025Outlier-Safe Pre-Training for Robust 4-Bit Quantization of Large Language Modelscited by 13
  7. 2023Stable Anisotropic Regularizationcited by 12
  8. 2025KV Cache Transform Coding for Compact Storage in LLM Inferencecited by 11
  9. 2026STAR-KV: Low-Rank KV Cache Compression via Soft Thresholding for Adaptive Rank Controlcited by 11
  10. 2026Next Concept Prediction in Discrete Latent Space Leads to Stronger Language Modelscited by 8
  11. 2026Training Transformers for KV Cache Compressibilitycited by 6
  12. 2026KV-CoRE: Benchmarking Data-Dependent Low-Rank Compressibility of KV-Caches in LLMscited by 5
  13. 2026NITP: Next Implicit Token Prediction for LLM Pre-trainingcited by 3
  14. 2026eOptShrinkQ: Near-Lossless KV Cache Compression Through Optimal Spectral Denoising and Quantizationcited by 2
  15. 2026ReQAT: Achieving Full-Precision Reasoning Accuracy with 4-bit Floating-Point Quantization-Aware Trainingcited by 2
  16. 2026Low-Rank Key Value Attentioncited by 1
  17. 2026Representation Without Reward: A JEPA Audit for LLM Fine-Tuningcited by 1
  18. 2026ATMA: Long-Context Language Modeling via Polar Attention and Gated-Delta Compression Memorycited by 1

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026Alignment Collapse Under KV Cache Quantization: Diagnosis and Mitigation
  2. 2025Tracing the Representation Geometry of Language Models from Pretraining to Post-training
  3. 2024MiniCache: KV Cache Compression in Depth Dimension for Large Language Models
  4. 2024KV Cache is 1 Bit Per Channel: Efficient Large Language Model Inference with Coupled Quantization
  5. 2026PuzzleKV: Page-Wise Low-Rank Decomposition for KV Cache Compression
  6. 2026High-accuracy Low-Bit KV-Cache Quantization via Local Distribution Restoration
  7. 2025KVSink: Understanding and Enhancing the Preservation of Attention Sinks in KV Cache Quantization for LLMs
  8. 2026Beyond Speedup -- Utilizing KV Cache for Sampling and Reasoning