# SSR: Sparse Segment Reduction for Ternary GEMM Acceleration

arXiv: https://arxiv.org/abs/2610.08403 · HTML: https://mlsanity.com/paper/2610.08403/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 9 of its 33 references that are in the map.
- First posted on arXiv 6 Oct 2026.

## Builds on (most cited first, 9 of 9)

- [The Era of 1-bit LLMs: All Large Language Models are in 1.58 Bits](https://mlsanity.com/paper/2402.17764/) (2024)
- [Trained Ternary Quantization](https://mlsanity.com/paper/1612.01064/) (2016)
- [Ternary Weight Networks](https://mlsanity.com/paper/1605.04711/) (2016)
- [Fast Matrix Multiplications for Lookup Table-Quantized LLMs](https://mlsanity.com/paper/2407.10960/) (2024)
- [Bitnet.cpp: Efficient Edge Inference for Ternary LLMs](https://mlsanity.com/paper/2502.11880/) (2025)
- [Learning Recurrent Binary/Ternary Weights](https://mlsanity.com/paper/1809.11086/) (2018)
- [Fast matrix multiplication for binary and ternary CNNs on ARM CPU](https://mlsanity.com/paper/2205.09120/) (2022)
- [An Efficient Matrix Multiplication Algorithm for Accelerating Inference in Binary and Ternary Neural Networks](https://mlsanity.com/paper/2411.06360/) (2024)
- [Large Language Models on Small Resource-Constrained Systems: Performance Characterization, Analysis and Trade-offs](https://mlsanity.com/paper/2412.15352/) (2024)

## Abstract (arXiv)

Large Language Models (LLMs) require substantial computational resources, limiting their deployment on resource-constrained hardware. Ternary LLMs mitigate these demands through weight quantization via ternary values, achieving significant compression often with 50-90% sparsity. However, existing approaches have limitations: methods optimized for ternary weights, such as BitNet, redundant segment reduction (RSR), and its improved version RSR++, do not exploit sparsity structures, while conventional sparse formats neglect ternary characteristics, foregoing dual optimization opportunities. In this paper, we introduce Sparse Segment Reduction (SSR), a ternary matrix multiplication method designed to accelerate the inference of ternary LLMs and general Ternary Weight Networks (TWNs). SSR has a dedicated optimized ternary data format and an algorithm that systematically exploits sparsity patterns through computation trees that scale with the sparsity. SSR provides theoretical gains with asymptotically faster inference than RSR++ for sparsity above 50%, while practical evaluations reveal performance improvements across all sparsity levels. Evaluation results show that SSR achieves 2.1-11.3x speedup over RSR++ on ternary GEMM with 45-95% sparsity. Furthermore, SSR achieves 3.5-6.3x end-to-end speedup and 4.9% of memory saving over RSR++ on the Llama-3 1B model inference.