mlsanityData to 14 Oct 2026

Empowering Hybrid Attention Models on NPUs

arXiv 2609.32114 ↗First posted 26 Sep 2026cs.DC cs.AIcited by 0

Not on the map. Why?

Abstract (arXiv)

Hybrid attention models have emerged as a crucial architecture for Large Language Models (LLMs) (e.g., the Qwen3.5 and Kimi series). Their memory and computational efficiency make them highly attractive for on-device inference, forming a promising synergy with edge Neural Processing Units (NPUs). However, naive execution of these hybrid models on edge NPUs fails to deliver these benefits, often bottlenecking the prefill stage due to severe memory-system inefficiencies and architectural mismatches within the linear attention (LA) layers. We present HA-NPU, the first system to enable efficient hybrid attention LLM inference on edge NPUs without modifying the underlying algorithms. HA-NPU enhances execution efficiency by reorganizing the dataflow of the LA components across three levels: (1) At the core level, it partitions workloads by the head dimension and fuses dependent operators, eliminating cross-core global memory accesses; (2) At the operator level, it reorders execution to consume intermediate tensors immediately, drastically minimizing local-buffer pressure; (3) At the tensor level, it employs dataflow-aware layout planning to minimize transformation overhead between matrix and vector processing units. Compared to competitive baselines, HA-NPU achieves up to 35.95× LA kernel speedup and 36.14× energy reduction, delivering up to 2.03× faster end-to-end request latency. The source code will be made publicly available at https://github.com/yinyuanzhang/HA-NPU

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 38 of its 70 references that are in the map. First posted on arXiv 26 Sep 2026.

Builds on

38 of 70 references.

References · 38

  1. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,944
  2. 2018HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answeringcited by 3,754
  3. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,673
  4. 2017TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehensioncited by 2,955
  5. 2023MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGIcited by 2,080
  6. 2020Transformers are RNNs: Fast Autoregressive Transformers with Linear Attentioncited by 2,022
  7. 2024MMLU-Pro: A More Robust and Challenging Multi-Task Language Understanding Benchmarkcited by 1,848
  8. 2017In-Datacenter Performance Analysis of a Tensor Processing Unitcited by 618
  9. 2023Retentive Network: A Successor to Transformer for Large Language Modelscited by 613
  10. 2024Gated Delta Networks: Improving Mamba2 with Delta Rulecited by 532
  11. 2023Gated Linear Attention Transformers with Hardware-Efficient Trainingcited by 514
  12. 2021Linear Transformers Are Secretly Fast Weight Programmerscited by 502
26 more
  1. 2024Parallelizing Linear Transformers with the Delta Rule over Sequence Lengthcited by 396
  2. 2023PowerInfer: Fast Large Language Model Serving with a Consumer-grade GPUcited by 217
  3. 2024Simple linear attention language models balance the recall-throughput tradeoffcited by 208
  4. 2023TPU v4: An Optically Reconfigurable Supercomputer for Machine Learning with Hardware Support for Embeddingscited by 202
  5. 2025Kimi Linear: An Expressive, Efficient Attention Architecturecited by 196
  6. 2024HGRN2: Gated Linear RNNs with State Expansioncited by 127
  7. 2020Data Movement Is All You Need: A Case Study on Optimizing Transformerscited by 108
  8. 2024Gated Slot Attention for Efficient Linear-Time Sequence Modelingcited by 76
  9. 2024Fast On-device LLM Inference with NPUscited by 62
  10. 2024T-MAC: CPU Renaissance via Table Lookup for Low-Bit LLM Deployment on Edgecited by 55
  11. 2026On the Design of Qwen3.8-Next Architecture: Evaluation, Efficiency, and Training Stabilitycited by 35
  12. 2026Gated DeltaNet-2: Decoupling Erase and Write in Linear Attentioncited by 33
  13. 2024Various Lengths, Constant Speed: Efficient Language Modeling with Lightning Attentioncited by 31
  14. 2018Optimizing CNN Model Inference on CPUscited by 29
  15. 2020Ordering Chaos: Memory-Aware Scheduling of Irregularly Wired Neural Networks for Edge Devicescited by 27
  16. 2024BlueLM-V-3B: Algorithm and System Co-Design for Multimodal Large Language Models on Mobile Devicescited by 25
  17. 2025Characterizing Mobile SoC for Accelerating Heterogeneous LLM Inferencecited by 22
  18. 2025MPK: A Compiler and Runtime for Mega-Kernelizing Tensor Programscited by 15
  19. 2024SmartMem: Layout Transformation Elimination and Adaptation for Efficient DNN Execution on Mobilecited by 12
  20. 2024Linear Transformers with Learnable Kernel Functions are Better In-Context Modelscited by 11
  21. 2025Agent.xpu: Efficient Scheduling of Agentic LLM Workloads on Heterogeneous SoCcited by 11
  22. 2025Scaling LLM Test-Time Compute with Mobile NPU on Smartphonescited by 11
  23. 2025Accelerating Mobile Language Model via Speculative Decoding and NPU-Coordinated Executioncited by 5
  24. 2025ShadowNPU: System and Algorithm Co-design for NPU-Centric On-Device LLM Inferencecited by 4
  25. 2025T-MAN: Enabling End-to-End Low-Bit LLM Inference on NPUs via Unified Table Lookupcited by 4
  26. 2025GenieBlue: Integrating both Linguistic and Multimodal Capabilities for Large Language Models on Mobile Devicescited by 3

Built on by

Not cited yet.