mlsanityData to 14 Oct 2026

Vectorization Of Narrow Matrix Multiplication for Ascend AI Inference Acceleration

arXiv 2609.16009 ↗First posted 3 Aug 2026cs.DCcited by 0

On the map: Efficient LLM inference and serving → Serving systems and request scheduling

Abstract (arXiv)

This research proposes and evaluates a novel approach to optimizing matrix multiplication (MatMul) on Huawei Ascend NPUs, motivated by a key insight: during matrix-vector multiplication (narrow MatMul), the Cube Unit (AIC) is often underutilized, while the Vector Unit (AIV) remains idle for most of the operator runtime. In this paper, we introduce the MatMul algorithm, which uses vector instructions of AscendC to effectively offload computations from the Cube Unit to the Vector Unit. The algorithm was tested and applied to accelerating the inference of MLA DeepSeek-V3 operator. By successfully overlapping AIV and AIC computations, our optimization showed a mean performance gain of 20% for a single token processing scenario. Our work addresses a significant gap in the literature on practical optimization techniques for AscendC, despite the availability of documentation and the active CANN community.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Serving systems and request scheduling, one of 12 directions in Efficient LLM inference and serving. Builds on 7 of its 15 references that are in the map. First posted on arXiv 3 Aug 2026.

Builds on

7 of 15 references.

Closest ancestors · 1

Same direction, newest first

  1. 2024Large Language Model Inference Acceleration: A Comprehensive Hardware Perspective6 Oct

Other references · 6

  1. 2017Attention Is All You Needcited by 60,361
  2. 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,703
  3. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,673
  4. 2017In-Datacenter Performance Analysis of a Tensor Processing Unitcited by 618
  5. 2021Understanding the Capabilities, Limitations, and Societal Impact of Large Language Modelscited by 160
  6. 2025Towards Efficient Multi-Scale Deformable Attention on NPUcited by 1

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026W4A16 Mixed-Precision Matrix Multiplication on Decoupled Architecture: Kernel Design and Memory Bottleneck Analysis for Ascend NPUs
  2. 2025The Cambrian Explosion of Mixed-Precision Matrix Multiplication for Quantized Deep Learning Inference
  3. 2026RSR-core: A High-Performance Engine for Low-Bit Matrix-Vector Multiplication
  4. 2025FlexiSAGA: A Flexible Systolic Array GEMM Accelerator for Sparse and Dense Processing
  5. 2022FullPack: Full Vector Utilization for Sub-Byte Quantized Inference on General Purpose CPUs
  6. 2021A Deep Learning Inference Scheme Based on Pipelined Matrix Multiplication Acceleration Design and Non-uniform Quantization
  7. 2023Stella Nera: A Differentiable Maddness-Based Hardware Accelerator for Efficient Approximate Matrix Multiplication
  8. 2026On the Limitations of Non-GPU AI Accelerators for Large-Model Inference: A Field Study of MoE and Multimodal Serving on Huawei Ascend