mlsanityData to 7 Oct 2026

Deja Vu: Contextual Sparsity for Efficient LLMs at Inference Time

arXiv 2310.17157 ↗First posted 26 Oct 2023cs.LGcited by 247

Not on the map. Why?

Cited by 66 in Efficient LLM inference and serving, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Large language models (LLMs) with hundreds of billions of parameters have sparked a new wave of exciting AI applications. However, they are computationally expensive at inference time. Sparsity is a natural approach to reduce this cost, but existing methods either require costly retraining, have to forgo LLM's in-context learning ability, or do not yield wall-clock time speedup on modern hardware. We hypothesize that contextual sparsity, which are small, input-dependent sets of attention heads and MLP parameters that yield approximately the same output as the dense model for a given input, can address these issues. We show that contextual sparsity exists, that it can be accurately predicted, and that we can exploit it to speed up LLM inference in wall-clock time without compromising LLM's quality or in-context learning ability. Based on these insights, we propose DejaVu, a system that uses a low-cost algorithm to predict contextual sparsity on the fly given inputs to each layer, along with an asynchronous and hardware-aware implementation that speeds up LLM inference. We validate that DejaVu can reduce the inference latency of OPT-175B by over 2X compared to the state-of-the-art FasterTransformer, and over 6X compared to the widely used Hugging Face implementation, without compromising model quality. The code is available at https://github.com/FMInference/DejaVu.

Key facts. Cited by 247 papers in the map (data through 7 Oct 2026). Builds on 84 of its 143 references that are in the map. First posted on arXiv 26 Oct 2023.

Builds on

84 of 143 references.

References · 84

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  4. 2015Distilling the Knowledge in a Neural Networkcited by 10,076
  5. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
  6. 2020Training data-efficient image transformers & distillation through attentioncited by 4,560
  7. 2021On the Opportunities and Risks of Foundation Modelscited by 3,961
  8. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,611
  9. 2016Pointer Sentinel Mixture Modelscited by 2,992
  10. 2019PIQA: Reasoning about Physical Commonsense in Natural Languagecited by 2,818
  11. 2017Billion-scale similarity search with GPUscited by 2,477
  12. 2015Deep Compression: Compressing Deep Neural Networks with Pruning, Trained Quantization and Huffman Codingcited by 2,291
38 more
  1. 2018The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networkscited by 2,250
  2. 2018Can a Suit of Armor Conduct Electricity? A New Dataset for Open Book Question Answeringcited by 2,114
  3. 2022GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformerscited by 1,838
  4. 2020Reformer: The Efficient Transformercited by 1,658
  5. 2017Quantization and Training of Neural Networks for Efficient Integer-Arithmetic-Only Inferencecited by 1,548
  6. 2022Rethinking the Role of Demonstrations: What Makes In-Context Learning Work?cited by 1,413
  7. 2022SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Modelscited by 1,338
  8. 2023SparseGPT: Massive Language Models Can Be Accurately Pruned in One-Shotcited by 1,090
  9. 2021An Explanation of In-context Learning as Implicit Bayesian Inferencecited by 915
  10. 2022LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scalecited by 881
  11. 2016Efficient and robust approximate nearest neighbor search using Hierarchical Navigable Small World graphscited by 853
  12. 2019Are Sixteen Heads Really Better than One?cited by 803
  13. 2018SNIP: Single-shot Network Pruning based on Connection Sensitivitycited by 802
  14. 2022GPT-NeoX-20B: An Open-Source Autoregressive Language Modelcited by 711
  15. 2018Rethinking the Value of Network Pruningcited by 595
  16. 2022Efficiently Scaling Transformer Inferencecited by 529
  17. 2021Sparsity in Deep Learning: Pruning and growth for efficient inference and training in neural networkscited by 521
  18. 2022ZeroQuant: Efficient and Affordable Post-Training Quantization for Large-Scale Transformerscited by 517
  19. 2022DeepSpeed Inference: Enabling Efficient Inference of Transformer Models at Unprecedented Scalecited by 403
  20. 2019On the Efficacy of Knowledge Distillationcited by 377
  21. 2018Filter Pruning via Geometric Median for Deep Convolutional Neural Networks Accelerationcited by 365
  22. 2019Data-Free Quantization Through Weight Equalization and Bias Correctioncited by 328
  23. 2022Data Distributional Properties Drive Emergent In-Context Learning in Transformerscited by 303
  24. 2014Sketching as a Tool for Numerical Linear Algebracited by 285
  25. 2020The Hardware Lotterycited by 160
  26. 2021Revisiting ResNets: Improved Training and Scaling Strategiescited by 158
  27. 2017The Shattered Gradients Problem: If resnets are the answer, then what is the question?cited by 156
  28. 2020Pre-training Tasks for Embedding-based Large-scale Retrievalcited by 153
  29. 2019Distilling Task-Specific Knowledge from BERT into Simple Neural Networkscited by 152
  30. 2019Improving Neural Network Quantization without Retraining using Outlier Channel Splittingcited by 150
  31. 2021Scatterbrain: Unifying Sparse and Low-rank Attention Approximationcited by 128
  32. 2020Data Movement Is All You Need: A Case Study on Optimizing Transformerscited by 108
  33. 2015Practical and Optimal LSH for Angular Distancecited by 105
  34. 2023Fast Attention Requires Bounded Entriescited by 102
  35. 2019What Can ResNet Learn Efficiently, Going Beyond Kernels?cited by 91
  36. 2020TurboTransformers: An Efficient GPU Serving System For Transformer Modelscited by 71
  37. 2019Real-Time Open-Domain Question Answering with Dense-Sparse Phrase Indexcited by 64
  38. 2022Rethinking the Role of Scale for In-Context Learning: An Interpretability-based Case Study at 66 Billion Scalecited by 64

50 most cited of 84 All 84 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 10; 2024: 116; 2025: 70; 2026: 51.

247 papers.

Sort:

By area:

  1. 2024MInference 1.0: Accelerating Pre-filling for Long-Context LLMs via Dynamic Sparse Attentioncited by 375
  2. 2025Small Language Models are the Future of Agentic AIcited by 254
  3. 2024The Unreasonable Ineffectiveness of the Deeper Layerscited by 219
  4. 2024CacheBlend: Fast Large Language Model Serving for RAG with Cached Knowledge Fusioncited by 211
  5. 2023PowerInfer: Fast Large Language Model Serving with a Consumer-grade GPUcited by 210
  6. 2024Video-XL: Extra-Long Vision Language Model for Hour-Scale Video Understandingcited by 210
  7. 2023Efficient Large Language Models: A Surveycited by 183
  8. 2023LLM in a flash: Efficient Large Language Model Inference with Limited Memorycited by 170
  9. 2023A Language Agent for Autonomous Drivingcited by 161
  10. 2024LLM Inference Unveiled: Survey and Roofline Model Insightscited by 147
  11. 2024The Remarkable Robustness of LLMs: Stages of Inference?cited by 138
  12. 2024Cut the Crap: An Economical Communication Pipeline for LLM-based Multi-Agent Systemscited by 129
38 more
  1. 2024Mobile Edge Intelligence for Large Language Models: A Contemporary Surveycited by 125
  2. 2025Mixture-of-Recursions: Learning Dynamic Recursive Depths for Adaptive Token-Level Computationcited by 120
  3. 2023SparQ Attention: Bandwidth-Efficient LLM Inferencecited by 119
  4. 2024Hydra: Sequentially-Dependent Draft Heads for Medusa Decodingcited by 113
  5. 2023Towards Efficient Generative Large Language Model Serving: A Survey from Algorithms to Systemscited by 101
  6. 2024On-Device Language Models: A Comprehensive Reviewcited by 101
  7. 2024TriForce: Lossless Acceleration of Long Sequence Generation with Hierarchical Speculative Decodingcited by 98
  8. 2024SLEB: Streamlining LLMs through Redundancy Verification and Elimination of Transformer Blockscited by 96
  9. 2025KVzip: Query-Agnostic KV Cache Compression with Context Reconstructioncited by 89
  10. 2024A Survey of Resource-efficient LLM and Multimodal Foundation Modelscited by 84
  11. 2023Punica: Multi-Tenant LoRA Servingcited by 80
  12. 2024Differentially Private Synthetic Data via Foundation Model APIs 2: Textcited by 74
  13. 2024Relaxed Recursive Transformers: Effective Parameter Sharing with Layer-wise LoRAcited by 72
  14. 2024Fiddler: CPU-GPU Orchestration for Fast Inference of Mixture-of-Experts Modelscited by 69
  15. 2024SWIFT: On-the-Fly Self-Speculative Decoding for LLM Inference Accelerationcited by 69
  16. 2024ReLU2 Wins: Discovering Efficient Activation Functions for Sparse LLMscited by 66
  17. 2024HOBBIT: A Mixed Precision Expert Offloading System for Fast MoE Inferencecited by 53
  18. 2024Training-Free Activation Sparsity in Large Language Modelscited by 51
  19. 2024MaskLLM: Learnable Semi-Structured Sparsity for Large Language Modelscited by 49
  20. 2025Do Language Models Use Their Depth Efficiently?cited by 48
  21. 2024ProSparse: Introducing and Enhancing Intrinsic Activation Sparsity within Large Language Modelscited by 43
  22. 2024CATS: Contextually-Aware Thresholding for Sparsity in Large Language Modelscited by 41
  23. 2024Memory3: Language Modeling with Explicit Memorycited by 39
  24. 2024Densing Law of LLMscited by 39
  25. 2024Dissecting Language Models: Machine Unlearning via Selective Pruningcited by 38
  26. 2024ProMoE: Fast MoE-based LLM Serving using Proactive Cachingcited by 38
  27. 2025The Cost of Dynamic Reasoning: Demystifying AI Agents and Test-Time Scaling from an AI Infrastructure Perspectivecited by 38
  28. 2024Turbo Sparse: Achieving LLM SOTA Performance with Minimal Activated Parameterscited by 37
  29. 2024Large Language Model Inference Acceleration: A Comprehensive Hardware Perspectivecited by 36
  30. 2024LC-LLM: Explainable Lane-Change Intention and Trajectory Predictions with Large Language Modelscited by 35
  31. 2024Decoding Speculative Decodingcited by 33
  32. 2024How Far Are We From AGI: Are LLMs All We Need?cited by 31
  33. 2024Zeroth-Order Fine-Tuning of LLMs with Extreme Sparsitycited by 27
  34. 2024InfiniPot: Infinite Context Processing on Memory-Constrained LLMscited by 27
  35. 2024WAGLE: Strategic Weight Attribution for Effective and Modular Unlearning in Large Language Modelscited by 25
  36. 2024Q-Sparse: All Large Language Models can be Fully Sparsely-Activatedcited by 24
  37. 2023TypeFly: Flying Drones with Large Language Modelcited by 23
  38. 2024Implicit Multimodal Alignment: On the Generalization of Frozen LLMs to Multimodal Inputscited by 22

50 most cited of 247 All 247 (JSON)