XFP: Quality-Targeted Adaptive Codebook Quantization with Sparse Outlier Separation for LLM Inference
On the map: Mixture-of-experts → MoE large language models
Abstract (arXiv)
We introduce XFP, a dynamic weight quantizer for LLM inference that inverts the conventional workflow: the operator specifies reconstruction quality floors on per-channel cosine similarity (one strict floor for attention and shared experts, one lazy floor for routed-expert MoE); XFP determines codebook size, outlier budget, and packing per layer automatically -- no Hessian, no calibration data, no manual bit-width selection. Each weight matrix is decomposed into a sparse fp16 outlier residual and a dense sub-byte index tensor into a per-group learned codebook. Two storage modes share one auto-select frontend and one fused decode kernel: V2 (per-channel Lloyd) and V2a (shared library of L=32 codebooks per layer). On Qwen3.5-122B-A10B under V2, XFP reaches 138 tok/s single-stream decode on workstation hardware (RTX PRO 6000 Blackwell, TP=2) at 94.49% GSM8K strict-match (3 seeds, n=3957), and is 49% faster than Marlin INT4 at TP=1. For models that do not fit in the target memory envelope, we present the H-Process: a quality-driven iteration over the two cosine thresholds that finds the operating point at which the model just fits while still producing sensible output. Three constraints define its search space: the operator-set thresholds, an OOM boundary at quantize-on-load, and a garbage boundary in generation (cosine similarity steers; benches verify). On Qwen3.5-397B-A17B (512 routed experts/layer), the H-Process fits the full expert population into 2x96 GB at ~3.4 effective bits and delivers 100.9 tok/s long-output decode at 66.72% GSM8K strict-match on the full 1319-problem set (single seed at submission; multi-seed evaluation in progress), exceeding INT4 with routed-expert pruning on memory, throughput, and accuracy simultaneously.
Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in MoE large language models, one of 12 directions in Mixture-of-experts. Builds on 11 of its 23 references that are in the map. First posted on arXiv 14 May 2026.
Builds on
11 of 23 references.
References · 11
- 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,944
- 2023QLoRA: Efficient Finetuning of Quantized LLMscited by 3,418
- 2022GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformerscited by 1,919
- 2022LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scalecited by 906
- 2023QuIP: 2-Bit Quantization of Large Language Models With Guaranteescited by 414
- 2024QuIP#: Even Better LLM Quantization with Hadamard Incoherence and Lattice Codebookscited by 378
- 2023SpQR: A Sparse-Quantized Representation for Near-Lossless LLM Weight Compressioncited by 367
- 2023SqueezeLLM: Dense-and-Sparse Quantizationcited by 294
- 2024Extreme Compression of Large Language Models via Additive Quantizationcited by 232
- 2024QTIP: Quantization with Trellises and Incoherence Processingcited by 91
- 2023Optimize Weight Rounding via Signed Gradient Descent for the Quantization of LLMscited by 50
Built on by
Not cited yet.
Similar, not cited
Close in wording, no citation either way
- 2025ICQuant: Index Coding enables Low-bit LLM Quantization
- 2026CubicQuant: Parametric Non-Uniform Codebooks for High-Throughput LLM Inference with 1-8-Bit Weights
- 2026AAAC: Activation-Aware Adaptive Codebooks for 4-bit LLM Weight Quantization
- 2024Taming Sensitive Weights : Noise Perturbation Fine-tuning for Robust LLM Quantization
- 2026MXSens: Sensitivity-Aware Mixed-Precision Quantization for Efficient LLM Inference
- 2025CodeGEMM: A Codebook-Centric Approach to Efficient GEMM in Quantized LLMs
- 2026MosaicQuant: Inlier-Outlier Disaggregation for Unified 4-Bit LLM Quantization
- 2026FlashQuant: Sparse-Dense Fusion for Memory-Efficient Outlier-Aware LLM Inference