mlsanityData to 7 Oct 2026

A-IO: Adaptive Inference Orchestration for Memory-Bound NPUs

arXiv 2604.09752 ↗First posted 10 Apr 2026cs.DC cs.AIcited by 0

Not on the map. Why?

Abstract (arXiv)

During the deployment of Large Language Models (LLMs), the autoregressive decoding phase on heterogeneous NPU platforms (e.g., Ascend 910B) faces severe memory-bound challenges. This study reveals the ``Model Scaling Paradox'' caused by the static deployment of single-sized models. It also points out the kernel synchronization overhead of fine-grained speculative decoding citeleviathan2023fast, chen2023speculative under NPU computational graph compilation, and the severe limitations of purely relying on micro-level acceleration algorithms like Prompt LookUp Decoding (PLD)

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 7 of its 12 references that are in the map. First posted on arXiv 10 Apr 2026.

Builds on

7 of 12 references.

References · 7

  1. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,767
  2. 2022GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformerscited by 1,838
  3. 2022Fast Inference from Transformers via Speculative Decodingcited by 1,615
  4. 2023Accelerating Large Language Model Decoding with Speculative Samplingcited by 975
  5. 2022LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scalecited by 881
  6. 2024RouteLLM: Learning to Route LLMs with Preference Datacited by 633
  7. 2025Pangu Embedded: An Efficient Dual-system LLM Reasoner with Metacognitioncited by 61

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026CATS: Cascaded Adaptive Tree Speculation for Memory-Limited LLM Inference Acceleration
  2. 2025ASPD: Unlocking Adaptive Serial-Parallel Decoding by Exploring Intrinsic Parallelism in LLMs
  3. 2025SpecRouter: Adaptive Routing for Multi-Level Speculative Decoding in Large Language Models
  4. 2026AdaptiveSD A Stability-Aware, Runtime-Adaptive Speculative Decoding Framework with Multi-Policy Orchestration for CPU-Constrained LLM Inference
  5. 2025Scaling Up, Speeding Up: A Benchmark of Speculative Decoding for Efficient LLM Test-Time Scaling
  6. 2025AdaSD: Adaptive Speculative Decoding for Efficient Language Model Inference
  7. 2024Minions: Accelerating Large Language Model Inference with Aggregated Speculative Execution
  8. 2026Accelerating OpenPangu Inference on NPU via Speculative Decoding