mlsanityData to 7 Oct 2026

A Persistent-State Dataflow Accelerator for Memory-Bound Linear Attention Decode on FPGA

arXiv 2603.05931 ↗First posted 6 Mar 2026cs.AR cs.LGcited by 2

On the map: Efficient LLM inference and serving → KV cache optimization

Abstract (arXiv)

Gated DeltaNet (GDN) is a linear attention mechanism that replaces the growing KV cache with a fixed-size recurrent state. Hybrid LLMs like Qwen3-Next use 75% GDN layers and achieve competitive accuracy to attention-only models. However, at batch-1, GDN decode is memory-bound on GPUs since the full recurrent state must be round-tripped through HBM every token. We show that this bottleneck is architectural, not algorithmic, as all subquadratic sequence models exhibit arithmetic intensities below 1 FLOP/B at decode time, making them more memory-bound than standard Transformers. We present an FPGA accelerator that eliminates this bottleneck by holding the full 2 MB recurrent state persistently in on-chip BRAM, converting the workload from memory-bound to compute-bound. Our design fuses the GDN recurrence into a five-phase pipelined datapath that performs only one read and one write pass over each state matrix per token, exploits Grouped Value Attention for paired-head parallelism, and overlaps preparation, computation, and output storage via dataflow pipelining. We explore four design points on an AMD Alveo U55C using Vitis HLS, varying head-level parallelism from 2 to 16 value-heads per iteration. Our fastest configuration achieves 63 μs per token, 4.5× faster than the GPU reference on NVIDIA H100 PCIe. Post-implementation power analysis reports 9.96 W on-chip, yielding up to 60× greater energy efficiency per token decoded.

Key facts. Cited by 2 papers in the map (data through 7 Oct 2026). Placed in KV cache optimization, one of 12 directions in Efficient LLM inference and serving. Builds on 22 of its 26 references that are in the map. First posted on arXiv 6 Mar 2026.

Builds on

22 of 26 references.

Closest ancestors · 1

Same direction, newest first

  1. 2025FlashInfer: Efficient and Customizable Attention Engine for LLM Inference Serving2 Jan

Other references · 21

  1. 2017Attention Is All You Needcited by 59,932
  2. 2023Mamba: Linear-Time Sequence Modeling with Selective State Spacescited by 4,616
  3. 2022Fast Inference from Transformers via Speculative Decodingcited by 1,615
  4. 2024Transformers are SSMs: Generalized Models and Efficient Algorithms Through Structured State Space Dualitycited by 1,283
  5. 2019Fast Transformer Decoding: One Write-Head is All You Needcited by 700
  6. 2022Efficiently Scaling Transformer Inferencecited by 529
  7. 2024Gated Delta Networks: Improving Mamba2 with Delta Rulecited by 483
  8. 2024Parallelizing Linear Transformers with the Delta Rule over Sequence Lengthcited by 373
  9. 2023SqueezeLLM: Dense-and-Sparse Quantizationcited by 285
  10. 2024LLM Inference Unveiled: Survey and Roofline Model Insightscited by 147
  11. 2024FlightLLM: Efficient Large Language Model Inference with a Complete Mapping Flow on FPGAscited by 60
  12. 2016ESE: Efficient Speech Recognition Engine with Sparse LSTM on FPGAcited by 50
9 more
  1. 2022DFX: A Low-latency Multi-FPGA Appliance for Accelerating Transformer-based Text Generationcited by 46
  2. 2023Understanding the Potential of FPGA-Based Spatial Acceleration for Large Language Model Inferencecited by 29
  3. 2024MARCA: Mamba Accelerator with ReConfigurable Architecturecited by 13
  4. 2025WaferLLM: Large Language Model Inference at Wafer Scalecited by 13
  5. 2025LightMamba: Efficient Mamba Acceleration on FPGA with Quantization and Hardware Co-designcited by 11
  6. 2024Benchmarking Edge AI Platforms for High-Performance ML Inferencecited by 9
  7. 2025SpecMamba: Accelerating Mamba Inference on FPGA with Speculative Decodingcited by 5
  8. 2025LUT-LLM: Efficient Large Language Model Inference with Memory-based Computations on FPGAscited by 4
  9. 2026FlexLLM: Composable HLS Library for Flexible Hybrid LLM Accelerator Designcited by 3

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 2.

2 papers.

  1. 2026LoopQ: Quantization for Recursive Transformerscited by 4
  2. 2026DeltaLog: Deferred Materialization of Recurrent States for Linear Attention Decodingcited by 1

Similar, not cited

Close in wording, no citation either way

  1. 2026PersistentKV: Page-Aware Decode Scheduling for Long-Context LLM Serving on Commodity GPUs
  2. 2026Heterogeneous LLM Serving with General-Purpose Processing-Near-Memory for Retrieval-Based Sparse Attention
  3. 2026Why Gated DeltaNet Survives 4-Bit Quantization: NVFP4 W4A4 for the Recurrent Half of a Hybrid 27B LLM
  4. 2026AMMA: A Multi-Chiplet Memory-Centric Architecture for Low-Latency 1M Context Attention Serving
  5. 2026Analytical Provisioning for Attention-FFN Disaggregated LLM Serving under Stochastic Workloads
  6. 2023Gated Linear Attention Transformers with Hardware-Efficient Training
  7. 2026DAMP: Decay-Aware Mixed-Precision Recurrent-State Quantization
  8. 2026TreeWY: Speculative Verification for Gated DeltaNet Hybrids