Data to 14 Oct 2026

Loki: Low-rank Keys for Efficient Sparse Attention

arXiv 2406.02542 ↗First posted 4 Jun 2024cs.LGcited by 89

Not on the map. Why?

Cited by 51 in Efficient LLM inference and serving, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Inference on large language models (LLMs) can be expensive in terms of the compute and memory costs involved, especially when long sequence lengths are used. In particular, the self-attention mechanism used in LLM inference contributes significantly to these costs, which has sparked an interest in approximating the self-attention computation to reduce such costs. In this work, we propose to approximate self-attention by focusing on the dimensionality of key vectors computed in the attention block. Our analysis reveals that key vectors lie in a significantly lower-dimensional space, consistently across several datasets and models. Exploiting this observation, we propose Loki, a novel sparse attention method that ranks and selects tokens in the KV-cache based on attention scores computed in low-dimensional space. Our evaluations show that Loki is able to speed up the attention computation due to reduced data movement (load/store) and compute costs while maintaining the efficacy of the models better than other popular approximation methods.

Key facts. Cited by 89 papers in the map (data through 14 Oct 2026). Builds on 25 of its 35 references that are in the map. First posted on arXiv 4 Jun 2024.

Builds on

25 of 35 references.

References · 25

2017Attention Is All You Needcited by 60,361
2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,944
2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,673
2023Mistral 7Bcited by 3,164
2016Pointer Sentinel Mixture Modelscited by 3,096
2023Efficient Streaming Language Models with Attention Sinkscited by 2,186
2023Pythia: A Suite for Analyzing Large Language Models Across Training and Scalingcited by 1,786
2020Reformer: The Efficient Transformercited by 1,668
2017Quantization and Training of Neural Networks for Efficient Integer-Arithmetic-Only Inferencecited by 1,593

13 more

2019Generating Long Sequences with Sparse Transformerscited by 1,546
2020Linformer: Self-Attention with Linear Complexitycited by 1,474
2023LongBench: A Bilingual, Multitask Benchmark for Long Context Understandingcited by 1,417
2015Aligning Books and Movies: Towards Story-like Visual Explanations by Watching Movies and Reading Bookscited by 1,043
2024TinyLlama: An Open-Source Small Language Modelcited by 618
2023Model Tells You What to Discard: Adaptive KV Cache Compression for LLMscited by 474
2021A White Paper on Neural Network Quantizationcited by 472
2023Scissorhands: Exploiting the Persistence of Importance Hypothesis for LLM KV Cache Compression at Test Timecited by 439
2023ZeroQuant-V2: Exploring Post-training Quantization in LLMs from Comprehensive Study to Low Rank Compensationcited by 79
2021Memory-efficient Transformers via Top-k Attentioncited by 60
2021Sparse Attention with Linear Unitscited by 41
2021AxoNN: An asynchronous, message-driven parallel framework for extreme-scale deep learningcited by 15
2023A 4D Hybrid Algorithm to Scale Parallel Training to Thousands of GPUscited by 6

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 10; 2025: 36; 2026: 43.

89 papers.

Sort:

By area:

2024ShadowKV: KV Cache in Shadows for High-Throughput Long-Context LLM Inferencecited by 134
2024A Survey on Large Language Model Acceleration based on KV Cache Managementcited by 131
2024RetrievalAttention: Accelerating Long-Context LLM Inference via Vector Retrievalcited by 116
2024MagicPIG: LSH Sampling for Efficient LLM Generationcited by 111
2024MagicDec: Breaking the Latency-Throughput Tradeoff for Long Context Generation with Speculative Decodingcited by 91
2024ThinK: Thinner Key Cache by Query-Driven Pruningcited by 68
2025RocketKV: Accelerating Long-Context LLM Inference via Two-Stage KV Cache Compressioncited by 41
2025Tensor Product Attention Is All You Needcited by 31
2024HashAttention: Semantic Sparsity for Faster Inferencecited by 24
2024SCOPE: Optimizing Key-Value Cache Compression in Long-context Generationcited by 20
2025Lattice: Learning to Efficiently Compress the Memorycited by 19
2024Lexico: Extreme KV Cache Compression via Sparse Coding over Universal Dictionariescited by 17

38 more

2026Past- and Future-Informed KV Cache Policy with Salience Estimation in Autoregressive Video Diffusioncited by 16
2026Learning to Evict from Key-Value Cachecited by 14
2025BLASST: Dynamic BLocked Attention Sparsity via Softmax Thresholdingcited by 13
2025LoLA: Low-Rank Linear Attention With Sparse Cachingcited by 12
2025SALS: Sparse Attention in Latent Space for KV cache Compressioncited by 11
2024Router-Tuning: A Simple and Effective Approach for Enabling Dynamic-Depth in Transformerscited by 10
2025Hardware-Efficient Attention for Fast Decodingcited by 10
2025KQ-SVD: Compressing the KV Cache with Provable Guarantees on Attention Fidelitycited by 10
2025Kascade: A Practical Sparse Attention Method for Long-Context LLM Inferencecited by 10
2025SpeCache: Speculative Key-Value Caching for Efficient Generation of LLMscited by 9
2026FASA: Frequency-aware Sparse Attentioncited by 8
2026Value-Aware Stochastic KV Cache Eviction for Reasoning Modelscited by 8
2025A2ATS: Retrieval-Based KV Cache Reduction via Windowed Rotary Position Embedding and Query-Aware Vector Quantizationcited by 7
2025Mustafar: Promoting Unstructured Sparsity for KV Cache Pruning in LLM Inferencecited by 7
2025HATA: Trainable and Hardware-Efficient Hash-Aware Top-k Attention for Scalable Large Model Inferencecited by 7
2025SVDq: 1.25-bit and 410x Key Cache Compression for LLM Attentioncited by 6
2025GraphKV: Breaking the Static Selection Paradigm with Graph-Based KV Cache Evictioncited by 6
2025vAttention: Verified Sparse Attentioncited by 6
2025Exploiting Sparsity for Long Context Inference: Million Token Contexts on Commodity GPUscited by 5
2025Compact Attention: Exploiting Structured Spatio-Temporal Sparsity for Fast Video Generationcited by 5
2025KVSwap: Disk-aware KV Cache Offloading for Long-Context On-device Inferencecited by 5
2025Trellis: Learning to Compress Key-Value Memory in Attention Modelscited by 5
2026KV-CoRE: Benchmarking Data-Dependent Low-Rank Compressibility of KV-Caches in LLMscited by 5
2025SAS: Simulated Attention Scorecited by 4
2025Retrospective Sparse Attention for Efficient Long-Context Generationcited by 4
2025XQuant: Breaking the Memory Wall for LLM Inference with KV Cache Rematerializationcited by 4
2025Judge Q: Trainable Queries for Optimized Information Retention in KV Cache Evictioncited by 4
2025NOSA: Native and Offloadable Sparse Attentioncited by 4
2025Efficient Low Rank Attention for Long-Context Inference in Large Language Modelscited by 4
2025A Mathematical Theory of Top-k Sparse Attention via Total Variation Distancecited by 4
2025KV Admission: Learning What to Write for Efficient Long-Context LLM Inferencecited by 4
2026The Residual Stream Is All You Need: On the Redundancy of the KV Cache in Transformer Inferencecited by 4
2026Towards Efficient Large Language Model Serving: A Survey on System-Aware KV Cache Optimizationcited by 4
2025Compress, Gather, and Recompute: REFORMing Long-Context Processing in Transformerscited by 3
2025Alleviating Forgetfulness of Linear Attention by Hybrid Sparse Attention and Contextualized Learnable Token Evictioncited by 3
2026Adaptive Layer Selection for Layer-Wise Token Pruning in LLM Inferencecited by 3
2026SOCKET: SOft Collision Kernel EsTimator for Sparse Attentioncited by 3
2026QUOKA: Query-Oriented KV Selection For Efficient LLM Prefillcited by 3

50 most cited of 89 All 89 (JSON)

Similar, not cited

Close in wording, no citation either way