# Vectorization Of Narrow Matrix Multiplication for Ascend AI Inference Acceleration

arXiv: https://arxiv.org/abs/2609.16009 · HTML: https://mlsanity.com/paper/2609.16009/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Placed in Serving systems and request scheduling, one of 12 directions in Efficient LLM inference and serving.
- Builds on 7 of its 15 references that are in the map.
- First posted on arXiv 3 Aug 2026.

## Placed in

- Efficient LLM inference and serving / [Serving systems and request scheduling](https://mlsanity.com/inference/serving-systems-and-request-scheduling/)

## Builds on (most cited first, 7 of 7)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [RoFormer: Enhanced Transformer with Rotary Position Embedding](https://mlsanity.com/paper/2104.09864/) (2021)
- [FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness](https://mlsanity.com/paper/2205.14135/) (2022)
- [In-Datacenter Performance Analysis of a Tensor Processing Unit](https://mlsanity.com/paper/1704.04760/) (2017)
- [Understanding the Capabilities, Limitations, and Societal Impact of Large Language Models](https://mlsanity.com/paper/2102.02503/) (2021)
- [Large Language Model Inference Acceleration: A Comprehensive Hardware Perspective](https://mlsanity.com/paper/2410.04466/) (2024)
- [Towards Efficient Multi-Scale Deformable Attention on NPU](https://mlsanity.com/paper/2505.14022/) (2025)

## Abstract (arXiv)

This research proposes and evaluates a novel approach to optimizing matrix multiplication (MatMul) on Huawei Ascend NPUs, motivated by a key insight: during matrix-vector multiplication (narrow MatMul), the Cube Unit (AIC) is often underutilized, while the Vector Unit (AIV) remains idle for most of the operator runtime. In this paper, we introduce the MatMul algorithm, which uses vector instructions of AscendC to effectively offload computations from the Cube Unit to the Vector Unit. The algorithm was tested and applied to accelerating the inference of MLA DeepSeek-V3 operator. By successfully overlapping AIV and AIC computations, our optimization showed a mean performance gain of 20% for a single token processing scenario. Our work addresses a significant gap in the literature on practical optimization techniques for AscendC, despite the availability of documentation and the active CANN community.