GLIDE: Guided Layerwise Hybrid Attention for Efficient LLM Inference
On the map: Efficient LLM inference and serving → KV cache optimization
Abstract (arXiv)
As Large Language Models scale to increasingly long contexts, the memory I/O and computational overhead of the Key-Value (KV) cache during decoding emerges as the primary throughput bottleneck. To address this, we propose GLIDE, a Guided Layerwise Hybrid Attention that strategically integrates sliding-window softmax attention with linear recurrent aggregation. GLIDE is motivated by layer-wise heterogeneity: early layers exhibit high sensitivity to softmax removal, while deeper layers demonstrate redundancy and tolerate aggressive replacement by linear alternatives. Leveraging this insight, GLIDE introduces a layer-wise adaptive mechanism wherein each layer balances an efficient linear recurrence with a variable-sized softmax window. Unlike uniform hybrid approaches, GLIDE non-uniformly compresses the softmax footprint across the model, reducing aggregate KV cache I/O while preserving expressive power where most vital. Empirical evaluations demonstrate the GLIDE achieves superior performance-efficiency tradeoffs, reducing end-to-end latency for long-context generation without compromising quality.
Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in KV cache optimization, one of 12 directions in Efficient LLM inference and serving. Builds on 20 of its 25 references that are in the map. First posted on arXiv 26 Jun 2026.
Builds on
20 of 25 references.
Closest ancestors · 6
Same direction, newest first
- 2024A Survey on Large Language Model Acceleration based on KV Cache Management27 Dec
- 2024Ada-KV: Optimizing KV Cache Eviction by Adaptive Budget Allocation for Efficient LLM Inference16 Jul
- 2024PyramidKV: Dynamic KV Cache Compression based on Pyramidal Information Funneling4 Jun
- 2024SnapKV: LLM Knows What You are Looking for Before Generation22 Apr
- 2024Get More with LESS: Synthesizing Recurrence with KV Cache Compression for Efficient LLM Inference14 Feb
- 2023H2O: Heavy-Hitter Oracle for Efficient Generative Inference of Large Language Models24 Jun
Other references · 14
- 2017Attention Is All You Needcited by 60,361
- 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
- 2020Longformer: The Long-Document Transformercited by 3,413
- 2023Mistral 7Bcited by 3,164
- 2020Transformers are RNNs: Fast Autoregressive Transformers with Linear Attentioncited by 2,022
- 2024Simple linear attention language models balance the recall-throughput tradeoffcited by 208
- 2024Leave No Context Behind: Efficient Infinite Context Transformers with Infini-attentioncited by 200
- 2024The Hedgehog & the Porcupine: Expressive Linear Attentions with Softmax Mimicrycited by 120
- 2024Bridging the Divide: Reconsidering Softmax and Linear Attentioncited by 48
- 2025Liger: Linearizing Large Language Models to Gated Recurrent Structurescited by 33
- 2022ViTALiTy: Unifying Low-rank and Sparse Approximation for Vision Transformer Acceleration with a Linear Taylor Attentioncited by 29
- 2024LoLCATs: On Low-Rank Linearizing of Large Language Modelscited by 27
Built on by
Not cited yet.
Similar, not cited
Close in wording, no citation either way
- 2026CateKV: On Sequential Consistency for Long-Context LLM Inference Acceleration
- 2026Flux Attention: Context-Aware Hybrid Attention for Efficient LLMs Inference
- 2026From Rigid to Dynamic: Entropy-Guided Adaptive Inference for Long-Context LLMs
- 2025DeltaLLM: Delta-Key Sparse Attention for Edge LLMs
- 2026LycheeDecode: Accelerating Long-Context LLM Inference via Hybrid-Head Sparse Decoding
- 2026BOSCH: Black-Box Binary Optimization for Short-Context Attention-Head Selection in LLMs
- 2026Double-P: Hierarchical Top-P Sparse Attention for Long-Context LLMs
- 2026Sliding-window beats linear attention