mlsanityData to 14 Oct 2026

RSR-core: A High-Performance Engine for Low-Bit Matrix-Vector Multiplication

arXiv 2603.27462 ↗First posted 29 Mar 2026cs.DS cs.LG cs.PFcited by 0

Not on the map. Why?

Abstract (arXiv)

Matrix-vector multiplication is a fundamental building block in neural networks, vector databases, and large language models, particularly during inference. As a result, efficient matrix-vector multiplication engines directly translate into more efficient inference. Recent work has explored low-bit quantization of model weights, where matrices are represented using binary (1-bit) or ternary (1.58-bit) values while activation is kept in higher precision. These representations enable efficient hardware-level computation. In parallel, algorithms such as Redundant Segment Reduction (RSR) provide theoretical guarantees for accelerating low-bit matrix-vector multiplication. However, existing implementations operate at the application level and cannot be efficiently integrated into hardware kernels, limiting practical performance. To bridge this gap, we present RSR-core, a high-performance engine that implements the RSR algorithm as optimized low-level kernels for both CPU and CUDA environments. RSR-core supports efficient matrix-vector multiplication for binary and ternary weight matrices and general vectors while enabling practical deployment of RSR algorithm in real inference pipelines. RSR-core is provided as a production-ready engine with HuggingFace integration for preprocessing low-bit models and running accelerated inference. Experimental results demonstrate significant performance improvements over baseline HuggingFace PyTorch multiplication, achieving up to 62x speedup on CPU and up to 1.9x speedup for token generation on CUDA for popular ternary LLMs. The source code is publicly available at https://github.com/UIC-InDeXLab/RSR-core.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 7 of its 9 references that are in the map. First posted on arXiv 29 Mar 2026.

Builds on

7 of 9 references.

References · 7

  1. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,305
  2. 2024The Era of 1-bit LLMs: All Large Language Models are in 1.58 Bitscited by 308
  3. 2023Towards Efficient Generative Large Language Model Serving: A Survey from Algorithms to Systemscited by 102
  4. 2021Neural ranking models for document retrievalcited by 16
  5. 2024An Efficient Matrix Multiplication Algorithm for Accelerating Inference in Binary and Ternary Neural Networkscited by 3
  6. 2025Maximum Inner Product is Query-Scaled Nearest Neighborcited by 2
  7. 2025Machine learning and high dimensional vector searchcited by 1

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2022FullPack: Full Vector Utilization for Sub-Byte Quantized Inference on General Purpose CPUs
  2. 2025BLaST: High Performance Inference and Pretraining using BLock Sparse Transformers
  3. 2025The Cambrian Explosion of Mixed-Precision Matrix Multiplication for Quantized Deep Learning Inference
  4. 2024Efficient Arbitrary Precision Acceleration for Large Language Models on GPU Tensor Cores
  5. 2024Accelerating a Triton Fused Kernel for W4A16 Quantized Inference with SplitK work decomposition
  6. 2025Transitive Array: An Efficient GEMM Accelerator with Result Reuse
  7. 2023Sparq: A Custom RISC-V Vector Processor for Efficient Sub-Byte Quantized Inference
  8. 2022LUT-GEMM: Quantized Matrix Multiplication based on LUTs for Efficient Inference in Large-Scale Generative Language Models