mlsanityData to 7 Oct 2026

Hardware-Software Co-design for 3D-DRAM-based LLM Serving Accelerator

arXiv 2603.04797 ↗First posted 5 Mar 2026cs.ARcited by 0

On the map: Efficient LLM inference and serving → Serving systems and request scheduling

Abstract (arXiv)

Large language models (LLMs) have been widely deployed for online generative services, where numerous LLM instances jointly handle workloads with fluctuating request arrival rates and variable request lengths. To efficiently execute coexisting compute-intensive and memory-intensive operators, near-memory processing (NMP) based computing paradigm has been extensively proposed. However, existing NMP designs adopt coarse-grained KV cache management and inflexible attention execution flow. Such limitations hinder these proposals from efficiently handling textithighly dynamic LLM serving workloads, limiting their ability to accelerate LLM serving. To tackle these problems, we propose Helios, a Hybrid-bonding-based ulineLLM ulineServing accelerator. Helios aims to bridge the fundamental gap between the dynamic nature of KV cache management in LLM serving and the distributed, non-uniform memory abstraction among NMP processing engines (PEs). To this end, we design both the intra-PE execution flow and the inter-PE communication primitives for distributed tiled attention execution. We further propose textitspatially-aware KV cache allocation mechanism to balance the attention workload distribution while minimizing the inter-PE data transfer overhead. Compared with existing GPU/NMP designs, Helios achieves 3.25 times (geomean) speedup and 3.36 times (geomean) better energy efficiency, along with up to 72%/76% P50/P99 time-between-tokens degradation.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Serving systems and request scheduling, one of 12 directions in Efficient LLM inference and serving. Builds on 34 of its 77 references that are in the map. First posted on arXiv 5 Mar 2026.

Builds on

34 of 77 references.

Closest ancestors · 4

Same direction, newest first

  1. 2025ServeGen: Workload Characterization and Generation of Large Language Model Serving in Production15 May
  2. 2025PIM Is All You Need: A CXL-Enabled GPU-Free System for Large Language Model Inference11 Feb
  3. 2024Llumnix: Dynamic Scheduling for Large Language Model Serving5 Jun
  4. 2024DistServe: Disaggregating Prefill and Decoding for Goodput-optimized Large Language Model Serving18 Jan

Other references · 30

  1. 2017Attention Is All You Needcited by 59,932
  2. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  3. 2025Qwen3 Technical Reportcited by 8,195
  4. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  5. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,767
  6. 2016Layer Normalizationcited by 5,579
  7. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,611
  8. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,464
  9. 2023FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioningcited by 2,551
  10. 2022Emergent Abilities of Large Language Modelscited by 2,483
  11. 2024Mixtral of Expertscited by 1,785
  12. 2020GLU Variants Improve Transformercited by 1,724
18 more
  1. 2019Root Mean Square Layer Normalizationcited by 1,425
  2. 2023GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpointscited by 1,311
  3. 2023SGLang: Efficient Execution of Structured Language Model Programscited by 1,107
  4. 2022CodeGen: An Open Large Language Model for Code with Multi-Turn Program Synthesiscited by 876
  5. 2019Fast Transformer Decoding: One Write-Head is All You Needcited by 700
  6. 2023Splitwise: Efficient generative LLM inference using phase splittingcited by 536
  7. 2022Efficiently Scaling Transformer Inferencecited by 529
  8. 2024FlashAttention-3: Fast and Accurate Attention with Asynchrony and Low-precisioncited by 507
  9. 2025FlashInfer: Efficient and Customizable Attention Engine for LLM Inference Servingcited by 248
  10. 2023TPU v4: An Optically Reconfigurable Supercomputer for Machine Learning with Hardware Support for Embeddingscited by 198
  11. 2018Online normalizer calculation for softmaxcited by 154
  12. 2024NeuPIMs: NPU-PIM Heterogeneous Acceleration for Batched LLM Inferencingcited by 48
  13. 2025Insights into DeepSeek-V3: Scaling Challenges and Reflections on Hardware for AI Architecturescited by 38
  14. 2024Duplex: A Device for Large Language Models with Mixture of Experts, Grouped Query Attention, and Continuous Batchingcited by 33
  15. 2023Ramulator 2.0: A Modern, Modular, and Extensible DRAM Simulatorcited by 26
  16. 2025PAPI: Exploiting Dynamic Parallelism in Large Language Model Decoding with a Processing-In-Memory-Enabled Computing Systemcited by 19
  17. 2021TENET: A Framework for Modeling Tensor Dataflow Based on Relation-centric Notationcited by 9
  18. 2025Stratum: System-Hardware Co-Design with Tiered Monolithic 3D-Stackable DRAM for Efficient MoE Servingcited by 9

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026RedKnot: Efficient Long-Context LLM Serving with Head-Aware KV Reuse and SegPagedAttention
  2. 2026SiliconBench: Speed, Memory, and Fidelity for LLM Serving on Unified-Memory Desktops
  3. 2025Systematic Exploration of Multi-core Architectures for Efficient LLM Serving using WaferAI-SIM
  4. 2024Tackling the Dynamicity in a Production LLM Serving System with SOTA Optimizations via Hybrid Prefill/Decode/Verify Scheduling on Efficient Meta-kernels
  5. 2026PAM: Processing Across Memory Hierarchy for Efficient KV-centric LLM Serving System
  6. 2026From Tensor Buffer to Distributed Memory Hierarchy: A Survey of KV Cache Management for LLM Serving
  7. 2025FastCache: Optimizing Multimodal LLM Serving through Lightweight KV-Cache Compression Framework
  8. 2024APEX: An Extensible and Dynamism-Aware Simulator for Automated Parallel Execution in LLM Serving