mlsanityData to 14 Oct 2026

Litespark Inference For CPUs: Ultra-Fast SIMD Framework for Ternary (1.58-bit) Language Models

arXiv 2605.06485 ↗First posted 7 May 2026cs.CL cs.AIcited by 0

Not on the map. Why?

Abstract (arXiv)

Large language models (LLMs) have transformed artificial intelligence, but their computational requirements remain prohibitive for most users. Standard inference demands expensive datacenter GPUs or cloud API access, leaving over one billion personal computers underutilized for AI workloads. Ternary models offer a path forward: their weights are constrained to {-1, 0, +1}, theoretically eliminating the need for floating-point multiplication. However, existing frameworks fail to exploit this structure, treating ternary models as dense floating-point networks. We address this gap with custom SIMD kernels that replace matrix multiplication with simple addition and subtraction operations, targeting the integer dot product instructions available on modern CPUs. Our implementation, Litespark-Inference, is pip-installable and integrates directly with Hugging-Face, achieving 18.15x higher throughput, 7.15x faster time-to-first-token and 6.03x memory reduction compared to standard PyTorch inference on Apple Silicon, with comparable or higher throughput speedups up to 95.81x on Intel and AMD processors.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 11 of its 23 references that are in the map. First posted on arXiv 7 May 2026.

Builds on

11 of 23 references.

References · 11

  1. 2017Attention Is All You Needcited by 60,361
  2. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,305
  3. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,673
  4. 2022GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformerscited by 1,919
  5. 2022SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Modelscited by 1,386
  6. 2022LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scalecited by 906
  7. 2021A Survey of Quantization Methods for Efficient Neural Network Inferencecited by 815
  8. 2015BinaryConnect: Training Deep Neural Networks with binary weights during propagationscited by 592
  9. 2024The Era of 1-bit LLMs: All Large Language Models are in 1.58 Bitscited by 308
  10. 2023BitNet: Scaling 1-bit Transformers for Large Language Modelscited by 189
  11. 2024T-MAC: CPU Renaissance via Table Lookup for Low-Bit LLM Deployment on Edgecited by 55

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026FairyFuse: Multiplication-Free LLM Inference on CPUs via Fused Ternary Kernels
  2. 2025Spectra 1.1: Scaling Laws and Efficient Inference for Ternary Language Models
  3. 2023Efficient LLM Inference on CPUs
  4. 2025TENET: An Efficient Sparsity-Aware LUT-Centric Architecture for Ternary LLM Inference On Edge
  5. 2024Spectra: Surprising Effectiveness of Pretraining Ternary Language Models at Scale
  6. 2024Highly Optimized Kernels and Fine-Grained Codebooks for LLM Inference on Arm CPUs
  7. 2026Hardware Generation and Exploration of Lookup Table-Based Accelerators for 1.58-bit LLM Inference
  8. 2026VitaLLM: A Versatile, Ultra-Compact Ternary LLM Accelerator with Dependency-Aware Scheduling ↗