# Context-Driven Performance Modeling for Causal Inference Operators on Neural Processing Units

arXiv: https://arxiv.org/abs/2509.25155 · HTML: https://mlsanity.com/paper/2509.25155/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 20 of its 28 references that are in the map.
- First posted on arXiv 29 Sep 2025.

## Builds on (most cited first, 20 of 20)

- [Mamba: Linear-Time Sequence Modeling with Selective State Spaces](https://mlsanity.com/paper/2312.00752/) (2023)
- [FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness](https://mlsanity.com/paper/2205.14135/) (2022)
- [Transformers are SSMs: Generalized Models and Efficient Algorithms Through Structured State Space Duality](https://mlsanity.com/paper/2405.21060/) (2024)
- [RWKV: Reinventing RNNs for the Transformer Era](https://mlsanity.com/paper/2305.13048/) (2023)
- [Retentive Network: A Successor to Transformer for Large Language Models](https://mlsanity.com/paper/2307.08621/) (2023)
- [Jamba: A Hybrid Transformer-Mamba Language Model](https://mlsanity.com/paper/2403.19887/) (2024)
- [Hyena Hierarchy: Towards Larger Convolutional Language Models](https://mlsanity.com/paper/2302.10866/) (2023)
- [FNet: Mixing Tokens with Fourier Transforms](https://mlsanity.com/paper/2105.03824/) (2021)
- [Hymba: A Hybrid-head Architecture for Small Language Models](https://mlsanity.com/paper/2411.13676/) (2024)
- [Fast On-device LLM Inference with NPUs](https://mlsanity.com/paper/2407.05858/) (2024)
- [Toeplitz Neural Network for Sequence Modeling](https://mlsanity.com/paper/2305.04749/) (2023)
- [Is Long Context All You Need? Leveraging LLM's Extended Context for NL2SQL](https://mlsanity.com/paper/2501.12372/) (2025)
- [Are Long-LLMs A Necessity For Long-Context Tasks?](https://mlsanity.com/paper/2405.15318/) (2024)
- [Designing Efficient LLM Accelerators for Edge Devices](https://mlsanity.com/paper/2408.00462/) (2024)
- [Benchmarking Edge AI Platforms for High-Performance ML Inference](https://mlsanity.com/paper/2409.14803/) (2024)
- [MNN-LLM: A Generic Inference Engine for Fast Large Language Model Deployment on Mobile Devices](https://mlsanity.com/paper/2506.10443/) (2025)
- [Fourier-Mixed Window Attention: Accelerating Informer for Long Sequence Time-Series Forecasting](https://mlsanity.com/paper/2307.00493/) (2023)
- [XAMBA: Enabling Efficient State Space Models on Resource-Constrained Neural Processing Units](https://mlsanity.com/paper/2502.06924/) (2025)
- [MambaLiteSR: Image Super-Resolution with Low-Rank Mamba using Knowledge Distillation](https://mlsanity.com/paper/2502.14090/) (2025)
- [Benchmarking the Performance of Large Language Models on the Cerebras Wafer Scale Engine](https://mlsanity.com/paper/2409.00287/) (2024)

## Abstract (arXiv)

The proliferation of large language models has driven demand for long-context inference on resource-constrained edge platforms. However, deploying these models on Neural Processing Units (NPUs) presents significant challenges due to architectural mismatch: the quadratic complexity of standard attention conflicts with NPU memory and compute patterns. This paper presents a comprehensive performance analysis of causal inference operators on a modern NPU, benchmarking quadratic attention against sub-quadratic alternatives including structured state-space models and causal convolutions. Our analysis reveals a spectrum of critical bottlenecks: quadratic attention becomes severely memory-bound with catastrophic cache inefficiency, while sub-quadratic variants span from compute-bound on programmable vector cores to memory-bound by data movement. These findings provide essential insights for co-designing hardware-aware models and optimization strategies to enable efficient long-context inference on edge platforms.