# Operator Fusion for LLM Inference on the Tensix Architecture

arXiv: https://arxiv.org/abs/2606.09879 · HTML: https://mlsanity.com/paper/2606.09879/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 6 of its 15 references that are in the map.
- First posted on arXiv 3 Jun 2026.

## Builds on (most cited first, 6 of 6)

- [LLM Inference Unveiled: Survey and Roofline Model Insights](https://mlsanity.com/paper/2402.16363/) (2024)
- [Accelerating stencils on the Tenstorrent Grayskull RISC-V accelerator](https://mlsanity.com/paper/2409.18835/) (2024)
- [Attention in SRAM on Tenstorrent Grayskull](https://mlsanity.com/paper/2407.13885/) (2024)
- [Assessing Tenstorrent's RISC-V MatMul Acceleration Capabilities](https://mlsanity.com/paper/2505.06085/) (2025)
- [Exploring Fast Fourier Transforms on the Tenstorrent Wormhole](https://mlsanity.com/paper/2506.15437/) (2025)
- [Accelerating Gravitational N-Body Simulations Using the RISC-V-Based Tenstorrent Wormhole](https://mlsanity.com/paper/2509.19294/) (2025)

## Abstract (arXiv)

This study addresses on-device inference bottlenecks of Transformer models on Tenstorrent's Tensix architecture and proposes an operator fusion strategy that enhances data locality. RMSNorm is fused with matrix multiplication in self-attention and in the FFN, enabling back-to-back execution of memory-bound and compute-bound operators in on-chip SRAM to significantly reduce DRAM reads/writes of intermediate results and scheduling overhead. To support multi-core parallelism, a NoC-based multicast mechanism is leveraged in which row/column master nodes efficiently distribute inputs and weights across the core mesh, alleviating DRAM bandwidth contention. Experiments on the Wormhole platform with Qwen2.5-0.5B, Qwen3-0.6B, and Qwen3-4B show up to 37.44% latency reduction for attention and 15.89% for MLP, with up to 7.91% reduction per decoder layer, while Pearson Correlation Coefficient (PCC) remains above 98.75%, confirming significant end-to-end efficiency gains under numerical consistency.