Context-Driven Performance Modeling for Causal Inference Operators on Neural Processing Units
Not on the map. Why?
Abstract (arXiv)
The proliferation of large language models has driven demand for long-context inference on resource-constrained edge platforms. However, deploying these models on Neural Processing Units (NPUs) presents significant challenges due to architectural mismatch: the quadratic complexity of standard attention conflicts with NPU memory and compute patterns. This paper presents a comprehensive performance analysis of causal inference operators on a modern NPU, benchmarking quadratic attention against sub-quadratic alternatives including structured state-space models and causal convolutions. Our analysis reveals a spectrum of critical bottlenecks: quadratic attention becomes severely memory-bound with catastrophic cache inefficiency, while sub-quadratic variants span from compute-bound on programmable vector cores to memory-bound by data movement. These findings provide essential insights for co-designing hardware-aware models and optimization strategies to enable efficient long-context inference on edge platforms.
Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 20 of its 28 references that are in the map. First posted on arXiv 29 Sep 2025.
Builds on
20 of 28 references.
References · 20
- 2023Mamba: Linear-Time Sequence Modeling with Selective State Spacescited by 4,718
- 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,673
- 2024Transformers are SSMs: Generalized Models and Efficient Algorithms Through Structured State Space Dualitycited by 1,336
- 2023RWKV: Reinventing RNNs for the Transformer Eracited by 827
- 2023Retentive Network: A Successor to Transformer for Large Language Modelscited by 613
- 2024Jamba: A Hybrid Transformer-Mamba Language Modelcited by 401
- 2023Hyena Hierarchy: Towards Larger Convolutional Language Modelscited by 393
- 2021FNet: Mixing Tokens with Fourier Transformscited by 347
- 2024Hymba: A Hybrid-head Architecture for Small Language Modelscited by 99
- 2024Fast On-device LLM Inference with NPUscited by 62
- 2023Toeplitz Neural Network for Sequence Modelingcited by 43
- 2025Is Long Context All You Need? Leveraging LLM's Extended Context for NL2SQLcited by 30
8 more
- 2024Are Long-LLMs A Necessity For Long-Context Tasks?cited by 22
- 2024Designing Efficient LLM Accelerators for Edge Devicescited by 13
- 2024Benchmarking Edge AI Platforms for High-Performance ML Inferencecited by 9
- 2025MNN-LLM: A Generic Inference Engine for Fast Large Language Model Deployment on Mobile Devicescited by 8
- 2023Fourier-Mixed Window Attention: Accelerating Informer for Long Sequence Time-Series Forecastingcited by 3
- 2025XAMBA: Enabling Efficient State Space Models on Resource-Constrained Neural Processing Unitscited by 3
- 2025MambaLiteSR: Image Super-Resolution with Low-Rank Mamba using Knowledge Distillationcited by 3
- 2024Benchmarking the Performance of Large Language Models on the Cerebras Wafer Scale Enginecited by 2
Built on by
Not cited yet.
Similar, not cited
Close in wording, no citation either way
- 2025RetroInfer: A Vector Storage Engine for Scalable Long-Context LLM Inference
- 2026An Efficient Hybrid Sparse Attention with CPU-GPU Parallelism for Long-Context Inference
- 2025Paged Attention Meets FlexAttention: Unlocking Long-Context Efficiency in Deployed Inference
- 2026STEEL: Sparsity-Aware Fused Attention for Energy-Efficient Long-Sequence Inference on AMD's XDNA NPU
- 2026Understand and Accelerate Memory Processing Pipeline for Large Language Model Inference
- 2025Polar Sparsity: High Throughput Batched LLM Inferencing with Scalable Contextual Sparsity
- 2025Reimagining Memory Access for LLM Inference: Compression-Aware Memory Controller Design
- 2024MAS-Attention: Memory-Aware Stream Processing for Attention Acceleration on Resource-Constrained Edge Devices