mlsanityData to 7 Oct 2026

FairyFuse: Multiplication-Free LLM Inference on CPUs via Fused Ternary Kernels

arXiv 2604.20913 ↗First posted 22 Apr 2026cs.LGcited by 0

Not on the map. Why?

Abstract (arXiv)

Large language models are increasingly deployed on CPU-only platforms where memory bandwidth is the primary bottleneck for autoregressive generation. Weight quantization to four bits or below reduces memory pressure, yet existing systems still dequantize weights and perform floating-point multiplications, limiting the achievable gains. Ternary weights in {-1, 0, +1} provide a more efficient alternative, replacing multiplications with conditional additions, subtractions, or no-ops. While Fairy2i shows that ternary LLMs can match FP16 quality, its runtime does not exploit this structure. We present FairyFuse, an inference system that enables multiplication-free execution on commodity CPUs by fusing the eight real-valued sub-GEMVs of each widely-linear layer into a single AVX-512 loop using masked additions and subtractions, with zero floating-point multiplications. Roofline analysis shows that 16x weight compression shifts memory-bound GEMV toward the compute regime on bandwidth-limited CPUs, yielding a 29.6x kernel speedup while offering little benefit on GPUs. End-to-end, FairyFuse achieves 32.4 tokens per second on a single Intel Xeon 8558P, outperforming llama.cpp Q4_K_M by 1.24x with near-lossless quality (WikiText-2 perplexity 5.52 vs. 5.47 FP16; downstream accuracy 66.0%).

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 30 of its 39 references that are in the map. First posted on arXiv 22 Apr 2026.

Builds on

30 of 39 references.

References · 30

  1. 2017Attention Is All You Needcited by 59,932
  2. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  3. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,767
  4. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,611
  5. 2016Pointer Sentinel Mixture Modelscited by 2,992
  6. 2015Deep Compression: Compressing Deep Neural Networks with Pruning, Trained Quantization and Huffman Codingcited by 2,291
  7. 2022GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformerscited by 1,838
  8. 2022SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Modelscited by 1,338
  9. 2016XNOR-Net: ImageNet Classification Using Binary Convolutional Neural Networkscited by 829
  10. 2022Efficiently Scaling Transformer Inferencecited by 529
  11. 2022ZeroQuant: Efficient and Affordable Post-Training Quantization for Large-Scale Transformerscited by 517
  12. 2022DeepSpeed Inference: Enabling Efficient Inference of Transformer Models at Unprecedented Scalecited by 403
18 more
  1. 2023QuIP: 2-Bit Quantization of Large Language Models With Guaranteescited by 397
  2. 2024QuIP#: Even Better LLM Quantization with Hadamard Incoherence and Lattice Codebookscited by 354
  3. 2023LLM-QAT: Data-Free Quantization Aware Training for Large Language Modelscited by 320
  4. 2024The Era of 1-bit LLMs: All Large Language Models are in 1.58 Bitscited by 289
  5. 2023SqueezeLLM: Dense-and-Sparse Quantizationcited by 285
  6. 2024Extreme Compression of Large Language Models via Additive Quantizationcited by 223
  7. 2017Deep Complex Networkscited by 215
  8. 2016Trained Ternary Quantizationcited by 203
  9. 2024QServe: W4A8KV4 Quantization and System Co-design for Efficient LLM Servingcited by 200
  10. 2016Ternary Weight Networkscited by 195
  11. 2024BiLLM: Pushing the Limit of Post-Training Quantization for LLMscited by 137
  12. 2024EfficientQAT: Efficient Quantization-Aware Training for Large Language Modelscited by 123
  13. 2023PB-LLM: Partially Binarized Large Language Modelscited by 80
  14. 2024BitDistiller: Unleashing the Potential of Sub-4-Bit LLMs via Self-Distillationcited by 80
  15. 2024OneBit: Towards Extremely Low-bit Large Language Modelscited by 77
  16. 2024T-MAC: CPU Renaissance via Table Lookup for Low-Bit LLM Deployment on Edgecited by 47
  17. 2025iFairy: the First 2-bit Complex LLM with All Parameters in ±1, ± icited by 5
  18. 2025Fairy2i: Training Complex LLMs from Real LLMs with All Parameters in ± 1, ± icited by 2

Built on by

Not cited yet.