mlsanityData to 14 Oct 2026

CoDec: Prefix-Shared Decoding Kernel for LLMs

arXiv 2505.17694 ↗First posted 23 May 2025cs.LGcited by 4

Not on the map. Why?

Abstract (arXiv)

Prefix-sharing among multiple prompts presents opportunities to combine the operations of the shared prefix, while attention computation in the decode stage, which becomes a critical bottleneck with increasing context lengths, is a memory-intensive process requiring heavy memory access on the key-value (KV) cache of the prefixes. Therefore, in this paper, we explore the potential of prefix-sharing in the attention computation of the decode stage. However, the tree structure of the prefix-sharing mechanism presents significant challenges for attention computation in efficiently processing shared KV cache access patterns while managing complex dependencies and balancing irregular workloads. To address the above challenges, we propose a dedicated attention kernel to combine the memory access of shared prefixes in the decoding stage, namely CoDec. CoDec delivers two key innovations: a novel shared-prefix attention kernel that optimizes memory hierarchy and exploits both intra-block and inter-block parallelism, and a comprehensive workload balancing mechanism that efficiently estimates cost, divides tasks, and schedules execution. Experimental results show that CoDec achieves an average 1.9× speedup and 120.9× memory access reduction compared to the state-of-the-art FlashDecoding kernel regarding attention computation in the decode stage and 3.8× end-to-end time per output token compared to the vLLM.

Key facts. Cited by 4 papers in the map (data through 14 Oct 2026). Builds on 33 of its 57 references that are in the map. First posted on arXiv 23 May 2025.

Builds on

33 of 57 references.

References · 33

  1. 2017Attention Is All You Needcited by 60,361
  2. 2020Language Models are Few-Shot Learnerscited by 32,419
  3. 2024The Llama 3 Herd of Modelscited by 14,715
  4. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,944
  5. 2023FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioningcited by 2,598
  6. 2023Code Llama: Open Foundation Models for Codecited by 2,139
  7. 2019Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelismcited by 1,868
  8. 2023GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpointscited by 1,346
  9. 2023SGLang: Efficient Execution of Structured Language Model Programscited by 1,141
  10. 2021NExT-QA:Next Phase of Question-Answering to Explaining Temporal Actionscited by 744
  11. 2019Fast Transformer Decoding: One Write-Head is All You Needcited by 717
  12. 2021Prompt Programming for Large Language Models: Beyond the Few-Shot Paradigmcited by 606
21 more
  1. 2023SpecInfer: Accelerating Generative Large Language Model Serving with Tree-based Speculative Inference and Verificationcited by 365
  2. 2023Large Language Models are Zero-Shot Rankers for Recommender Systemscited by 309
  3. 2015A Critical Review of Recurrent Neural Networks for Sequence Learningcited by 224
  4. 2024CacheBlend: Fast Large Language Model Serving for RAG with Cached Knowledge Fusioncited by 223
  5. 2023LooGLE: Can Long-Context Language Models Understand Long Contexts?cited by 202
  6. 2024Mooncake: A KVCache-centric Disaggregated Architecture for LLM Servingcited by 196
  7. 2023SARATHI: Efficient LLM Inference by Piggybacking Decodes with Chunked Prefillscited by 193
  8. 2021Sequence Parallelism: Long Sequence Training from System Perspectivecited by 126
  9. 2024LoongServe: Efficiently Serving Long-Context Large Language Models with Elastic Sequence Parallelismcited by 112
  10. 2023Evaluating Open-Domain Question Answering in the Era of Large Language Modelscited by 111
  11. 2024ChunkAttention: Efficient Self-Attention with Prefix-Aware KV Cache and Two-Phase Partitioncited by 82
  12. 2024Preble: Efficient Distributed Prompt Scheduling for LLM Servingcited by 70
  13. 2024MemServe: Context Caching for Disaggregated LLM Serving with Elastic Memory Poolcited by 68
  14. 2024ExeGPT: Constraint-Aware Resource Scheduling for LLM Inferencecited by 45
  15. 2025Cache-Craft: Managing Chunk-Caches for Efficient Retrieval-Augmented Generationcited by 36
  16. 2024BatchLLM: Optimizing Large Batched LLM Inference with Global Prefix Sharing and Throughput-oriented Token Batchingcited by 32
  17. 2024Context Parallelism for Scalable Million-Token Inferencecited by 26
  18. 2024DeFT: Decoding with Flash Tree-attention for Efficient Tree-structured LLM Inferencecited by 19
  19. 2024BlendServe: Optimizing Offline Inference for Auto-regressive Large Models with Resource-aware Batchingcited by 19
  20. 2024Anda: Unlocking Efficient LLM Inference with a Variable-Length Grouped Activation Data Formatcited by 9
  21. 2025Echo: Efficient Co-Scheduling of Hybrid Online-Offline Tasks for Large Language Model Servingcited by 3

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 2; 2026: 2.

4 papers.

  1. 2025MEPIC: Memory Efficient Position Independent Caching for LLM Servingcited by 11
  2. 2026Semantic Cache Distillation: Efficient State Transfer via Reuse and Selective Patchingcited by 5
  3. 2025TyphoonMLA: A Mixed Naive-Absorb MLA Kernel For Shared Prefixcited by 0
  4. 2026HiKV: Hierarchical Importance-Aware KV Cache with Hardware Acceleration for LLM Decodingcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025PAT: Accelerating LLM Decoding via Prefix-Aware Attention with Resource Efficient Multi-Tile Kernel
  2. 2025Hardware-Efficient Attention for Fast Decoding
  3. 2024Bifurcated Attention: Accelerating Massively Parallel Decoding with Shared Prefixes in LLMs
  4. 2026Where Activation Sparsity and KV-Cache Sparsity Cross in LLM Decoding
  5. 2026Salca: A Sparsity-Aware Hardware Accelerator for Efficient Long-Context Attention Decoding
  6. 2026SwiftKV: An Edge-Oriented Attention Algorithm and Multi-Head Accelerator for Fast, Efficient LLM Decoding
  7. 2026SAC: Disaggregated KV Cache System for Sparse Attention LLMs with CXL
  8. 2026Unifying Sparse Attention with Hierarchical Memory for Scalable Long-Context LLM Serving