mlsanityData to 7 Oct 2026

Challenges in Deploying Long-Context Transformers: A Theoretical Peak Performance Analysis

arXiv 2405.08944 ↗First posted 14 May 2024cs.LG cs.AI cs.CLcited by 44

On the map: LLM agents and tool use → Software engineering coding agents

also in 1 more
Abstract (arXiv)

Transformer-based long context generative models power emerging AI applications like hour-long video understanding and project-level coding agent. Deploying long context transformers (e.g., 100K to 10M tokens) is prohibitively expensive compared to short context (e.g., 4K tokens) model variants. Reducing the cost of long-context transformers is becoming a pressing research and engineering challenge starting from the year of 2024. This work describes a concurrent programming framework for quantitatively analyzing the efficiency challenges in serving multiple long-context requests under limited size of GPU high-bandwidth memory (HBM) regime. We give a detailed analysis of how all additional computational costs, compared to 4K context, trace back to textitone single source: the large size of the KV cache. We use a 34B GPT-3.5 level model of 50K context on A100 NVLink as a running example, and describe how its large KV cache causes four types of deployment challenges: (1) prefilling long inputs takes much longer compute time and GPU memory than short inputs; (2) after prefilling, the large KV cache residing on the GPU HBM substantially restricts the number of concurrent users being served; (3) during decoding, repeatedly reading the KV cache from HBM to SM largely increases latency; (4) when KV cache memory overflows, swapping it from HBM to DDR causes significant context switching latency. We use this framework to analyze existing works and identify possibilities of combining them to build end-to-end systems. Overall, this work offers a foundational framework for analyzing long context transformer deployment and identifies directions towards reducing the inference cost of 1M context to be as cheap as 4K.

Key facts. Cited by 44 papers in the map (data through 7 Oct 2026). Placed in Software engineering coding agents, one of 16 directions in LLM agents and tool use. Builds on 28 of its 31 references that are in the map. First posted on arXiv 14 May 2024.

Builds on

28 of 31 references.

References · 28

  1. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  2. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  3. 2020Scaling Laws for Neural Language Modelscited by 6,411
  4. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,767
  5. 2023Mamba: Linear-Time Sequence Modeling with Selective State Spacescited by 4,616
  6. 2023Qwen Technical Reportcited by 3,273
  7. 2024Mixtral of Expertscited by 1,785
  8. 2022Fast Inference from Transformers via Speculative Decodingcited by 1,615
  9. 2024DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Modelcited by 1,093
  10. 2023Accelerating Large Language Model Decoding with Speculative Samplingcited by 975
  11. 2024SnapKV: LLM Knows What You are Looking for Before Generationcited by 848
  12. 2019Analyzing Multi-Head Self-Attention: Specialized Heads Do the Heavy Lifting, the Rest Can Be Prunedcited by 844
16 more
  1. 2023H2O: Heavy-Hitter Oracle for Efficient Generative Inference of Large Language Modelscited by 805
  2. 2024Yi: Open Foundation Models by 01.AIcited by 706
  3. 2024KIVI: A Tuning-Free Asymmetric 2bit Quantization for KV Cachecited by 604
  4. 2023Model Tells You What to Discard: Adaptive KV Cache Compression for LLMscited by 466
  5. 2024LayerSkip: Enabling Early Exit Inference and Self-Speculative Decodingcited by 302
  6. 2022Confident Adaptive Language Modelingcited by 297
  7. 2021LongT5: Efficient Text-To-Text Transformer for Long Sequencescited by 232
  8. 2024Retrieval Head Mechanistically Explains Long-Context Factualitycited by 219
  9. 2024Data Engineering for Scaling Language Models to 128K Contextcited by 183
  10. 2024You Only Cache Once: Decoder-Decoder Architectures for Language Modelscited by 135
  11. 2024Dynamic Memory Compression: Retrofitting LLMs for Accelerated Inferencecited by 115
  12. 2024LongAlign: A Recipe for Long Context Alignment of Large Language Modelscited by 102
  13. 2024TriForce: Lossless Acceleration of Long Sequence Generation with Hierarchical Speculative Decodingcited by 98
  14. 2023CoLT5: Faster Long-Range Transformers with Conditional Computationcited by 84
  15. 2024Infinite-LLM: Efficient LLM Service for Long Context with DistAttention and Distributed KVCachecited by 71
  16. 2024WKVQuant: Quantizing Weight and Key/Value Cache for Large Language Models Gains Morecited by 69

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 15; 2025: 18; 2026: 11.

44 papers.

Sort:

By area:

  1. 2024MInference 1.0: Accelerating Pre-filling for Long-Context LLMs via Dynamic Sparse Attentioncited by 375
  2. 2024HippoRAG: Neurobiologically Inspired Long-Term Memory for Large Language Modelscited by 273
  3. 2025FlexPrefill: A Context-Aware Sparse Attention Mechanism for Efficient Long-Sequence Inferencecited by 121
  4. 2024A Simple and Effective L2 Norm-Based Strategy for KV Cache Compressioncited by 84
  5. 2025Expected Attention: KV Cache Compression by Estimating Attention from Future Queries Distributioncited by 70
  6. 2024Palu: Compressing KV-Cache with Low-Rank Projectioncited by 59
  7. 2024SCBench: A KV Cache-Centric Analysis of Long-Context Methodscited by 59
  8. 2024Block Transformer: Global-to-Local Language Modeling for Fast Inferencecited by 32
  9. 2025The Sparse Frontier: Sparse Attention Trade-offs in Transformer LLMscited by 28
  10. 2024Compute Or Load KV Cache? Why Not Both?cited by 25
  11. 2025SWAN-GPT: An Efficient and Scalable Approach for Long-Context Language Modelingcited by 16
  12. 2025FreeKV: Boosting KV Cache Retrieval for Efficient LLM Inferencecited by 14
32 more
  1. 2026KVzap: Fast, Adaptive, and Faithful KV Cache Pruningcited by 14
  2. 2025xKV: Cross-Layer KV-Cache Compression via Aligned Singular Vector Extractioncited by 13
  3. 2025SlimInfer: Accelerating Long-Context LLM Inference via Dynamic Token Pruningcited by 13
  4. 2024Analog In-Memory Computing Attention Mechanism for Fast and Energy-Efficient Large Language Modelscited by 11
  5. 2024Interpolating Video-LLMs: Toward Longer-sequence LMMs in a Training-free Mannercited by 10
  6. 2025Re-ttention: Ultra Sparse Visual Generation via Attention Statistical Reshapecited by 10
  7. 2026SideQuest: Model-Driven KV Cache Management for Long-Horizon Agentic Reasoningcited by 10
  8. 2025KQ-SVD: Compressing the KV Cache with Provable Guarantees on Attention Fidelitycited by 9
  9. 2026PackCache: A Training-Free Acceleration Method for Unified Autoregressive Video Generation via Compact KV-Cachecited by 9
  10. 2024Let the Code LLM Edit Itself When You Edit the Codecited by 8
  11. 2024Residual vector quantization for KV cache compression in large language modelcited by 7
  12. 2024freePruner: A Training-free Approach for Large Multimodal Model Accelerationcited by 7
  13. 2025An LMM for Efficient Video Understanding via Reinforced Compression of Video Cubescited by 7
  14. 2025CATransformers: Carbon Aware Transformers Through Joint Model-Hardware Optimizationcited by 7
  15. 2024StreamAdapter: Efficient Test Time Adaptation from Contextual Streamscited by 6
  16. 2024HashEvict: A Pre-Attention KV Cache Eviction Strategy using Locality-Sensitive Hashingcited by 6
  17. 2025SparK: Query-Aware Unstructured Sparsity with Recoverable KV Cache Channel Pruningcited by 6
  18. 2025Q-Filters: Leveraging QK Geometry for Efficient KV Cache Compressioncited by 4
  19. 2025Retrospective Sparse Attention for Efficient Long-Context Generationcited by 4
  20. 2026Scout Before You Attend: Sketch-and-Walk Sparse Attention for Efficient LLM Inferencecited by 4
  21. 2026Sparse Prefix Caching for Hybrid and Recurrent LLM Servingcited by 4
  22. 2025SALE : Low-bit Estimation for Efficient Sparse Attention in Long-context LLM Prefillingcited by 3
  23. 2025Learned Structure in Cartridges: Keys as Shareable Routers in Self-Studied Representationscited by 2
  24. 2026RAP: KV-Cache Compression via RoPE-Aligned Pruningcited by 2
  25. 2024Discourse Features Enhance Detection of Document-Level Machine-Generated Contentcited by 1
  26. 2025Towards Sampling Data Structures for Tensor Products in Turnstile Streamscited by 1
  27. 2025Small Language Models for Phishing Website Detection: Cost, Performance, and Privacy Trade-Offscited by 1
  28. 2026SLAY: Geometry-Aware Spherical Linearized Attention with Yat-Kernelcited by 1
  29. 2026VSPrefill: Vertical-Slash Sparse Attention with Lightweight Indexing for Long-Context Prefillingcited by 1
  30. 2026Think Before you Write: QA-Guided Reasoning for Character Descriptions in Bookscited by 0
  31. 2026Salca: A Sparsity-Aware Hardware Accelerator for Efficient Long-Context Attention Decodingcited by 0
  32. 2026OVAL: Output-Aware Local Page Bases for KV Cache Retrievalcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Characterizing State Space Model and Hybrid Language Model Performance with Long Context
  2. 2025End-to-End Transformer Acceleration Through Processing-in-Memory Architectures
  3. 2025EdgeInfinite: A Memory-Efficient Infinite-Context Transformer for Edge Devices
  4. 2024A Little Goes a Long Way: Efficient Long Context Training and Inference with Partial Contexts
  5. 2026Long-Context Aware Upcycling: A New Frontier for Hybrid LLM Scaling
  6. 2025Compress, Gather, and Recompute: REFORMing Long-Context Processing in Transformers
  7. 2025Context Discipline and Performance Correlation: Analyzing LLM Performance and Quality Degradation Under Varying Context Lengths
  8. 2025Modality Agnostic Efficient Long Range Encoder