mlsanityData to 7 Oct 2026

SFMP: Fine-Grained, Hardware-Friendly and Search-Free Mixed-Precision Quantization for Large Language Models

arXiv 2602.01027 ↗First posted 1 Feb 2026cs.LGcited by 1

Not on the map. Why?

Abstract (arXiv)

Mixed-precision quantization is a promising approach for compressing large language models under tight memory budgets. However, existing mixed-precision methods typically suffer from one of two limitations: they either rely on expensive discrete optimization to determine precision allocation, or introduce hardware inefficiencies due to irregular memory layouts. We propose SFMP, a search-free and hardware-friendly mixed-precision quantization framework for large language models. The framework is built upon four novel ideas: Fractional bit-width, which extends integer bit-width for weight matrix to fractional value and transforms discrete precision allocation as a continuous problem; 2)Block-wise mixed-precision, enabling fine-grained precision within weight matrices while remaining hardware-friendly; 3)Row-column weight reordering, which aggregates salient weights via row and column reordering, incurring only a small activation reordering overhead during inference; 4)Unified GEMM kernel, which supports mixed-precision GEMM at arbitrary average bit-width. Extensive experiments demonstrate that SFMP outperforms state-of-the-art layer-wise mixed-precision methods under the same memory constraints, while significantly reducing quantization cost and improving inference efficiency. Code is available at https://github.com/Nkniexin/SFMP

Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Builds on 40 of its 62 references that are in the map. First posted on arXiv 1 Feb 2026.

Builds on

40 of 62 references.

References · 40

  1. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  2. 2024The Llama 3 Herd of Modelscited by 14,192
  3. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  4. 2025Qwen3 Technical Reportcited by 8,195
  5. 2020Measuring Massive Multitask Language Understandingcited by 7,768
  6. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,710
  7. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,018
  8. 2016Pointer Sentinel Mixture Modelscited by 2,992
  9. 2019PIQA: Reasoning about Physical Commonsense in Natural Languagecited by 2,818
  10. 2019BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questionscited by 2,140
  11. 2022SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Modelscited by 1,338
  12. 2023AWQ: Activation-aware Weight Quantization for LLM Compression and Accelerationcited by 1,234
28 more
  1. 2024SpinQuant: LLM quantization with learned rotationscited by 446
  2. 2023SqueezeLLM: Dense-and-Sparse Quantizationcited by 285
  3. 2019HAWQ-V2: Hessian Aware trace-Weighted Quantization of Neural Networkscited by 250
  4. 2024BiLLM: Pushing the Limit of Post-Training Quantization for LLMscited by 137
  5. 2024EfficientQAT: Efficient Quantization-Aware Training for Large Language Modelscited by 123
  6. 2022LUT-GEMM: Quantized Matrix Multiplication based on LUTs for Efficient Inference in Large-Scale Generative Language Modelscited by 98
  7. 2024SliM-LLM: Salience-Driven Mixed-Precision Quantization for Large Language Modelscited by 61
  8. 2022Mixed-Precision Neural Network Quantization via Learned Layer-wise Importancecited by 53
  9. 2024T-MAC: CPU Renaissance via Table Lookup for Low-Bit LLM Deployment on Edgecited by 47
  10. 2025ParetoQ: Improving Scaling Laws in Extremely Low-bit LLM Quantizationcited by 44
  11. 2025Sustainable LLM Inference for Edge AI: Evaluating Quantized LLMs for Energy Efficiency, Output Accuracy, and Inference Latencycited by 32
  12. 2024ShiftAddLLM: Accelerating Pretrained LLMs via Post-Training Multiplication-Less Reparameterizationcited by 27
  13. 2020BiQGEMM: Matrix Multiplication with Lookup Table For Binary-Coding-based Quantized DNNscited by 17
  14. 2025AMQ: Enabling AutoML for Mixed-precision Weight-Only Quantization of Large Language Modelscited by 15
  15. 2023Fast and Efficient 2-bit LLM Inference on GPU: 2/4/16-bit in a Weight Matrix with Asynchronous Dequantizationcited by 13
  16. 2024Channel-Wise Mixed-Precision Quantization for Large Language Modelscited by 11
  17. 2024BitStack: Any-Size Compression of Large Language Models in Variable Memory Environmentscited by 9
  18. 2025Mix-QViT: Mixed-Precision Vision Transformer Quantization Driven by Layer Importance and Quantization Sensitivitycited by 9
  19. 2025FGMP: Fine-Grained Mixed-Precision Weight and Activation Quantization for Hardware-Accelerated LLM Inferencecited by 9
  20. 2023DeepGEMM: Accelerated Ultra Low-Precision Inference on CPU Architectures using Lookup Tablescited by 8
  21. 2024LRP-QViT: Mixed-Precision Vision Transformer Quantization via Layer-wise Relevance Propagationcited by 8
  22. 2025BlockDialect: Block-wise Fine-grained Mixed Format Quantization for Energy-Efficient LLM Inferencecited by 8
  23. 2025FIGLUT: An Energy-Efficient Accelerator Design for FP-INT GEMM Using Look-Up Tablescited by 8
  24. 2025AnyBCQ: Hardware Efficient Flexible Binary-Coded Quantization for Multi-Precision LLMscited by 8
  25. 2024Edge Intelligence Optimization for Large Language Model Inference with Batching and Quantizationcited by 7
  26. 2025FlexQuant: A Flexible and Efficient Dynamic Precision Switching Framework for LLM Quantizationcited by 5
  27. 2024OutlierTune: Efficient Channel-Wise Quantization for Large Language Modelscited by 4
  28. 2025DILEMMA: Joint LLM Quantization and Distributed LLM Inference Over Edge Computing Systemscited by 2

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 1.

1 paper.

  1. 2026AlignQuant: Tile-Aligned Mixed-Precision Quantization for Efficient LLM Generationcited by 0