# At-the-Roofline Sparse Tensor Contractions on Vector Processors for Transformer Inference

arXiv: https://arxiv.org/abs/2607.25504 · HTML: https://mlsanity.com/paper/2607.25504/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 13 of its 19 references that are in the map.
- First posted on arXiv 28 Jul 2026.

## Builds on (most cited first, 13 of 13)

- [The Llama 3 Herd of Models](https://mlsanity.com/paper/2407.21783/) (2024)
- [Scaling Laws for Generative Mixed-Modal Language Models](https://mlsanity.com/paper/2301.03728/) (2023)
- [Dynamic Sparse No Training: Training-Free Fine-tuning for Sparse LLMs](https://mlsanity.com/paper/2310.08915/) (2023)
- [Training-Free Activation Sparsity in Large Language Models](https://mlsanity.com/paper/2408.14690/) (2024)
- [Wanda++: Pruning Large Language Models via Regional Gradients](https://mlsanity.com/paper/2503.04992/) (2025)
- [SparseLLM: Towards Global Pruning for Pre-trained Language Models](https://mlsanity.com/paper/2402.17946/) (2024)
- [Toward Adaptive Large Language Models Structured Pruning via Hybrid-grained Weight Importance Assessment](https://mlsanity.com/paper/2403.10799/) (2024)
- [ShadowLLM: Predictor-based Contextual Sparsity for Large Language Models](https://mlsanity.com/paper/2406.16635/) (2024)
- [Stream Semantic Registers: A Lightweight RISC-V ISA Extension Achieving Full Compute Utilization in Single-Issue Cores](https://mlsanity.com/paper/1911.08356/) (2019)
- [GVSoC: A Highly Configurable, Fast and Accurate Full-Platform Simulator for RISC-V based IoT Processors](https://mlsanity.com/paper/2201.08166/) (2022)
- [DuoGPT: Training-free Dual Sparsity through Activation-aware Pruning in LLMs](https://mlsanity.com/paper/2506.20194/) (2025)
- [Spatz: Clustering Compact RISC-V-Based Vector Units to Maximize Computing Efficiency](https://mlsanity.com/paper/2309.10137/) (2023)
- [Optimizing Structured-Sparse Matrix Multiplication in RISC-V Vector Processors](https://mlsanity.com/paper/2501.10189/) (2025)

## Abstract (arXiv)

Fine-grained weight pruning and activation sparsification have emerged as effective approaches for reducing the compute and memory cost of inference for Transformer models. In the moderate-sparsity regime, Gustavson's dataflow provides a natural execution model for exploiting both activation and weight sparsity on vector processors through metadata-driven indexed accumulation. However, existing RVV architectures lack native support for this pattern, forcing kernels to rely on software index decoding and L1-backed indexed memory operations that keep sparse tensor contractions far below their roofline performance bound. We present Ventaglio, a runtime-configurable sparse execution unit coupled with RVV ISA extensions that drives sparse tensor contractions toward their roofline through indexed gather-accumulate-scatter support. Integrated into an open-source vector processing cluster and implemented in 12nm FinFET, Ventaglio accelerates sparse tensor contraction kernels by 6.9--7.4× over optimized RVV baselines, with only 3.1% area overhead for a cluster of tightly-L1 coupled vector processing elements. We build a performance-accurate instruction-level model of the Ventaglio extension, calibrate it against RTL implementation, and leverage it for scale-out performance analysis on a large 4×4 multi-cluster system. Using a DuoGPT-pruned LLaMA-3-8B model with practical 40--60% dual sparsity, Ventaglio achieves 2.40--5.25× and 2.06--3.16× speedup over dense baselines during prefill and autoregressive decoding, respectively.