mlsanityData to 7 Oct 2026

Hydragen: High-Throughput LLM Inference with Shared Prefixes

arXiv 2402.05099 ↗First posted 7 Feb 2024cs.LGcited by 71

Not on the map. Why?

Cited by 41 in Efficient LLM inference and serving, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Transformer-based large language models (LLMs) are now deployed to hundreds of millions of users. LLM inference is commonly performed on batches of sequences that share a prefix, such as few-shot examples or a chatbot system prompt. Decoding in this large-batch setting can be bottlenecked by the attention operation, which reads large key-value (KV) caches from memory and computes inefficient matrix-vector products for every sequence in the batch. In this work, we introduce Hydragen, a hardware-aware exact implementation of attention with shared prefixes. Hydragen computes attention over the shared prefix and unique suffixes separately. This decomposition enables efficient prefix attention by batching queries together across sequences, reducing redundant memory reads and enabling the use of hardware-friendly matrix multiplications. Our method can improve end-to-end CodeLlama-13b throughput by up to 32x against competitive baselines, with speedup growing with the batch size and shared prefix length. Hydragen also enables the use of very long shared contexts: with a large batch size, increasing the prefix length from 1K to 16K tokens decreases Hydragen throughput by less than 15%, while the throughput of baselines drops by over 90%. Hydragen generalizes beyond simple prefix-suffix decomposition and can be applied to tree-based prompt sharing patterns, allowing us to further reduce inference time on competitive programming problems by 55%.

Key facts. Cited by 71 papers in the map (data through 7 Oct 2026). Builds on 17 of its 26 references that are in the map. First posted on arXiv 7 Feb 2024.

Builds on

17 of 26 references.

References · 17

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  3. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  4. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,767
  5. 2022Self-Consistency Improves Chain of Thought Reasoning in Language Modelscited by 5,601
  6. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,611
  7. 2023Tree of Thoughts: Deliberate Problem Solving with Large Language Modelscited by 3,155
  8. 2023FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioningcited by 2,551
  9. 2022Competition-Level Code Generation with AlphaCodecited by 1,427
  10. 2023GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpointscited by 1,311
  11. 2021Measuring Coding Challenge Competence With APPScited by 879
  12. 2023Graph of Thoughts: Solving Elaborate Problems with Large Language Modelscited by 873
5 more
  1. 2019Fast Transformer Decoding: One Write-Head is All You Needcited by 700
  2. 2023FlexGen: High-Throughput Generative Inference of Large Language Models with a Single GPUcited by 567
  3. 2022DeepSpeed Inference: Enabling Efficient Inference of Transformer Models at Unprecedented Scalecited by 403
  4. 2021Self-attention Does Not Need O(n2) Memorycited by 152
  5. 2023FlashFFTConv: Efficient Convolutions for Long Sequences with Tensor Corescited by 32

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 23; 2025: 19; 2026: 29.

71 papers.

Sort:

By area:

  1. 2024Large Language Monkeys: Scaling Inference Compute with Repeated Samplingcited by 946
  2. 2025FlashInfer: Efficient and Customizable Attention Engine for LLM Inference Servingcited by 248
  3. 2024Certifiably Robust RAG against Retrieval Corruptioncited by 144
  4. 2024From Decoding to Meta-Generation: Inference-time Algorithms for Large Language Modelscited by 136
  5. 2024ShadowKV: KV Cache in Shadows for High-Throughput Long-Context LLM Inferencecited by 132
  6. 2024A Survey on Large Language Model Acceleration based on KV Cache Managementcited by 125
  7. 2025Cartridges: Lightweight and general-purpose long context representations via self-studycited by 74
  8. 2024MoE-Lightning: High-Throughput MoE Inference on Memory-constrained GPUscited by 73
  9. 2024Preble: Efficient Distributed Prompt Scheduling for LLM Servingcited by 68
  10. 2024SCBench: A KV Cache-Centric Analysis of Long-Context Methodscited by 59
  11. 2024Training of Physical Neural Networkscited by 47
  12. 2024TurboSpec: Closed-loop Speculation Control System for Optimizing LLM Serving Goodputcited by 36
38 more
  1. 2024A Simple Model of Inference Scaling Lawscited by 36
  2. 2024BatchLLM: Optimizing Large Batched LLM Inference with Global Prefix Sharing and Throughput-oriented Token Batchingcited by 31
  3. 2024Compute Or Load KV Cache? Why Not Both?cited by 25
  4. 2024Context Parallelism for Scalable Million-Token Inferencecited by 25
  5. 2024Optimizing LLM Queries in Relational Data Analytics Workloadscited by 20
  6. 2024DeFT: Decoding with Flash Tree-attention for Efficient Tree-structured LLM Inferencecited by 19
  7. 2024InstInfer: In-Storage Attention Offloading for Cost-Effective Long-Context LLM Inferencecited by 19
  8. 2024BlendServe: Optimizing Offline Inference for Auto-regressive Large Models with Resource-aware Batchingcited by 19
  9. 2025Auditing Prompt Caching in Language Model APIscited by 19
  10. 2026Don't Break the Cache: An Evaluation of Prompt Caching for Long-Horizon Agentic Taskscited by 18
  11. 2025ETS: Efficient Tree Search for Inference-Time Scalingcited by 15
  12. 2025Accurate KV Cache Quantization with Outlier Tokens Tracingcited by 15
  13. 2025Generative AI Act II: Test Time Scaling Drives Cognition Engineeringcited by 14
  14. 2026TraceLab: Characterizing Coding Agent Workloads for LLM Servingcited by 14
  15. 2026CONCUR: High-Throughput Agentic Batch Inference of LLM via Congestion-Based Concurrency Controlcited by 13
  16. 2025Kinetics: Rethinking Test-Time Scaling Lawscited by 11
  17. 2025HyGen: Efficient LLM Serving via Elastic Online-Offline Request Co-locationcited by 10
  18. 2026PackCache: A Training-Free Acceleration Method for Unified Autoregressive Video Generation via Compact KV-Cachecited by 9
  19. 2024KVDirect: Distributed Disaggregated LLM Inferencecited by 8
  20. 2024Bifurcated Attention: Accelerating Massively Parallel Decoding with Shared Prefixes in LLMscited by 7
  21. 2024New Solutions on LLM Acceleration, Optimization, and Applicationcited by 7
  22. 2025ElasticMM: Efficient Multimodal LLMs Serving with Elastic Multimodal Parallelismcited by 7
  23. 2025Efficient Speculative Decoding for Llama at Scale: Challenges and Solutionscited by 7
  24. 2026Execution-State Capsules: Graph-Bound Execution-State Checkpoint and Restore for Low-Latency, Small-Batch, On-Device Physical-AI Servingcited by 7
  25. 2026HYPIC: Accelerating Hybrid-Attention LLM Serving with Position-Independent Cachingcited by 7
  26. 2025Optimal Scheduling Algorithms for LLM Inference: Theory and Practicecited by 5
  27. 2025Queueing, Predictions, and LLMs: Challenges and Open Problemscited by 4
  28. 2024Accelerating Direct Preference Optimization with Prefix Sharingcited by 3
  29. 2025FastCache: Optimizing Multimodal LLM Serving through Lightweight KV-Cache Compression Frameworkcited by 3
  30. 2025MoE-Lens: Towards the Hardware Limit of High-Throughput MoE LLM Serving Under Resource Constraintscited by 3
  31. 2024Efficient Encoder-Decoder Transformer Decoding for Decomposable Taskscited by 2
  32. 2025Learned Structure in Cartridges: Keys as Shareable Routers in Self-Studied Representationscited by 2
  33. 2025xGR: Efficient Generative Recommendation Serving at Scalecited by 2
  34. 2025SpecMemo: Speculative Decoding is in Your Pocketcited by 1
  35. 2025CaliDrop: KV Cache Compression with Calibrationcited by 1
  36. 2026Joint Encoding of KV-Cache Blocks for Scalable LLM Servingcited by 1
  37. 2026PrefixWall: Mitigating Prefix Caching Side Channels in Shared LLM Systemscited by 1
  38. 2026Layer-wise MoE Routing Locality under Shared-Prefix Code Generation: Token-Identity Decomposition and Compile-Equivalent Fork Redundancycited by 1

50 most cited of 71 All 71 (JSON)