mlsanityData to 14 Oct 2026

SwiftKV: An Edge-Oriented Attention Algorithm and Multi-Head Accelerator for Fast, Efficient LLM Decoding

arXiv 2601.10953 ↗First posted 16 Jan 2026cs.ARcited by 0

On the map: Efficient LLM inference and serving → KV cache optimization

Abstract (arXiv)

Edge acceleration for large language models is crucial for their widespread application; however, achieving fast attention inference and efficient decoding on resource-constrained edge accelerators remains challenging. This paper presents SwiftKV Attention, a per-token pipelined, low-latency single-pass attention inference algorithm, where every (kt, vt) in the KV cache is processed exactly once in a uniform per-token pipeline without score materialization, blockwise softmax, or a second pass, thereby enabling fast execution on edge accelerators with a single hardware set and no resource-intensive parallelism. Furthermore, to address the limited support for multi-head LLM decoding in existing accelerators, we design the SwiftKV-MHA accelerator, which enables high precision attention and low precision GEMV on the same processor array, achieving fast and efficient multi-head parallel decoding. Experimental results show that, on the edge accelerator, the SwiftKV Attention algorithm achieves a 7.16* speedup over native attention and significantly outperforms other attention algorithms. SwiftKV-MHA further reduces attention latency by 13.48*; under the same settings, it improves generation speed by 17.4% and increases token efficiency by 1.98* compared with state-of-the-art works.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in KV cache optimization, one of 12 directions in Efficient LLM inference and serving. Builds on 10 of its 19 references that are in the map. First posted on arXiv 16 Jan 2026.

Builds on

10 of 19 references.

References · 10

  1. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,673
  2. 2022SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Modelscited by 1,386
  3. 2023Ring Attention with Blockwise Transformers for Near-Infinite Contextcited by 457
  4. 2018Online normalizer calculation for softmaxcited by 162
  5. 2024FlightLLM: Efficient Large Language Model Inference with a Complete Mapping Flow on FPGAscited by 64
  6. 2022DFX: A Low-latency Multi-FPGA Appliance for Accelerating Transformer-based Text Generationcited by 48
  7. 2024POD-Attention: Unlocking Full Prefill-Decode Overlap for Faster LLM Inferencecited by 47
  8. 2024EdgeLLM: A Highly Efficient CPU-FPGA Heterogeneous Edge Accelerator for Large Language Modelscited by 21
  9. 2023ITA: An Energy-Efficient Attention and Softmax Accelerator for Quantized Transformerscited by 12
  10. 2025Pushing up to the Limit of Memory Bandwidth and Capacity Utilization for Efficient LLM Decoding on Embedded FPGAcited by 7

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025PAT: Accelerating LLM Decoding via Prefix-Aware Attention with Resource Efficient Multi-Tile Kernel
  2. 2025Hardware-Efficient Attention for Fast Decoding
  3. 2026HiKV: Hierarchical Importance-Aware KV Cache with Hardware Acceleration for LLM Decoding
  4. 2026Salca: A Sparsity-Aware Hardware Accelerator for Efficient Long-Context Attention Decoding
  5. 2026OasisKV: Scaling In-Decode KV Cache Beyond HBM with Lookahead Sparse Prefetching
  6. 2026VitaLLM: A Versatile and Tiny Accelerator for Mixed-Precision LLM Inference on Edge Devices
  7. 2024DeFT: Decoding with Flash Tree-attention for Efficient Tree-structured LLM Inference
  8. 2025DeltaLLM: Delta-Key Sparse Attention for Edge LLMs