mlsanityData to 7 Oct 2026

Compressing Context to Enhance Inference Efficiency of Large Language Models

arXiv 2310.06201 ↗First posted 9 Oct 2023cs.CLcited by 185

Not on the map. Why?

Cited by 69 in Efficient LLM inference and serving, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Large language models (LLMs) achieved remarkable performance across various tasks. However, they face challenges in managing long documents and extended conversations, due to significantly increased computational requirements, both in memory and inference time, and potential context truncation when the input exceeds the LLM's fixed context length. This paper proposes a method called Selective Context that enhances the inference efficiency of LLMs by identifying and pruning redundancy in the input context to make the input more compact. We test our approach using common data sources requiring long context processing: arXiv papers, news articles, and long conversations, on tasks of summarisation, question answering, and response generation. Experimental results show that Selective Context significantly reduces memory cost and decreases generation latency while maintaining comparable performance compared to that achieved when full context is used. Specifically, we achieve a 50% reduction in context cost, resulting in a 36% reduction in inference memory usage and a 32% reduction in inference time, while observing only a minor drop of .023 in BERTscore and .038 in faithfulness on four downstream applications, indicating that our method strikes a good balance between efficiency and performance.

Key facts. Cited by 185 papers in the map (data through 7 Oct 2026). Builds on 14 of its 25 references that are in the map. First posted on arXiv 9 Oct 2023.

Builds on

14 of 25 references.

References · 14

  1. 2017Attention Is All You Needcited by 59,932
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  4. 2020Measuring Massive Multitask Language Understandingcited by 7,768
  5. 2019BERTScore: Evaluating Text Generation with BERTcited by 5,462
  6. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,710
  7. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,464
  8. 2020Longformer: The Long-Document Transformercited by 3,362
  9. 2023Sparks of Artificial General Intelligence: Early experiments with GPT-4cited by 2,579
  10. 2019Generating Long Sequences with Sparse Transformerscited by 1,523
  11. 2023Learning to Compress Prompts with Gist Tokenscited by 379
  12. 2020Asking and Answering Questions to Evaluate the Factual Consistency of Summariescited by 376
2 more
  1. 2023Adapting Language Models to Compress Contextscited by 350
  2. 2023Unlocking Context Constraints of LLMs: Enhancing Context Efficiency of LLMs with Self-Information-Based Content Filteringcited by 62

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 1; 2024: 40; 2025: 57; 2026: 87.

185 papers.

Sort:

By area:

  1. 2024MInference 1.0: Accelerating Pre-filling for Long-Context LLMs via Dynamic Sparse Attentioncited by 375
  2. 2025R1-Searcher: Incentivizing the Search Capability in LLMs via Reinforcement Learningcited by 322
  3. 2024Quiet-STaR: Language Models Can Teach Themselves to Think Before Speakingcited by 292
  4. 2025MemAgent: Reshaping Long-Context LLM with Multi-Conv RL-based Memory Agentcited by 235
  5. 2025MEM1: Learning to Synergize Memory and Reasoning for Efficient Long-Horizon Agentscited by 226
  6. 2024A Survey on Efficient Inference for Large Language Modelscited by 197
  7. 2025LightMem: Lightweight and Efficient Memory-Augmented Generationcited by 163
  8. 2024ShadowKV: KV Cache in Shadows for High-Throughput Long-Context LLM Inferencecited by 132
  9. 2024Mobile Edge Intelligence for Large Language Models: A Contemporary Surveycited by 125
  10. 2024xRAG: Extreme Context Compression for Retrieval-augmented Generation with One Tokencited by 123
  11. 2025FlexPrefill: A Context-Aware Sparse Attention Mechanism for Efficient Long-Sequence Inferencecited by 121
  12. 2025Lifelong Learning of Large Language Model based Agents: A Roadmapcited by 78
38 more
  1. 2024Beyond the Limits: A Survey of Techniques to Extend the Context Length in Large Language Modelscited by 71
  2. 2024LazyLLM: Dynamic Token Pruning for Efficient Long Context LLM Inferencecited by 70
  3. 2025Process vs. Outcome Reward: Which is Better for Agentic RAG Reinforcement Learningcited by 68
  4. 2024SCBench: A KV Cache-Centric Analysis of Long-Context Methodscited by 59
  5. 2025Scaling LLM Multi-turn RL with End-to-end Summarization-based Context Managementcited by 51
  6. 2024Efficient Prompting Methods for Large Language Models: A Surveycited by 46
  7. 2024DeciMamba: Exploring the Length Extrapolation Potential of Mambacited by 45
  8. 2025Reducing Cost of LLM Agents with Trajectory Reductioncited by 44
  9. 2024Auto-RAG: Autonomous Retrieval-Augmented Generation for Large Language Modelscited by 41
  10. 2026SWE-Pruner: Self-Adaptive Context Pruning for Coding Agentscited by 40
  11. 2024SampleAttention: Near-Lossless Acceleration of Long Context LLM Inference with Adaptive Structured Sparse Attentioncited by 39
  12. 2024Discovering the Gems in Early Layers: Accelerating Long-Context LLMs with 1000x Input Token Reductioncited by 39
  13. 2024Towards Evaluating and Building Versatile Large Language Models for Medicinecited by 38
  14. 2025LongCodeZip: Compress Long Context for Code Language Modelscited by 38
  15. 2025A Survey of Multimodal Retrieval-Augmented Generationcited by 37
  16. 2025Making Slow Thinking Faster: Compressing LLM Chain-of-Thought via Step Entropycited by 37
  17. 2024Block-Attention for Efficient Prefillingcited by 34
  18. 2025ReARTeR: Retrieval-Augmented Reasoning with Trustworthy Process Rewardingcited by 32
  19. 2024NovelQA: Benchmarking Question Answering on Documents Exceeding 200K Tokenscited by 31
  20. 2025R1-Searcher++: Incentivizing the Dynamic Knowledge Acquisition of LLMs via Reinforcement Learningcited by 30
  21. 2026SkillReducer: Optimizing LLM Agent Skills for Token Efficiencycited by 29
  22. 2024C-RAG: Certified Generation Risks for Retrieval-Augmented Language Modelscited by 27
  23. 2023Fewer is More: Boosting LLM Reasoning with Reinforced Context Pruningcited by 26
  24. 2024A Controlled Study on Long Context Extension and Generalization in LLMscited by 26
  25. 2025SePer: Measure Retrieval Utility Through The Lens Of Semantic Perplexity Reductioncited by 24
  26. 2024Multi-Layer Transformers Gradient Can be Approximated in Almost Linear Timecited by 22
  27. 2024TACO-RL: Task Aware Prompt Compression Optimization with Reinforcement Learningcited by 22
  28. 2025A Survey of Token Compression for Efficient Multimodal Large Language Modelscited by 22
  29. 2025Vision-centric Token Compression in Large Language Modelcited by 21
  30. 2025PISCO: Pretty Simple Compression for Retrieval-Augmented Generationcited by 20
  31. 2025GMSA: Enhancing Context Compression via Group Merging and Layer Semantic Alignmentcited by 20
  32. 2024A Tighter Complexity Analysis of SparseGPTcited by 19
  33. 2024FMint: Bridging Human Designed and Data Pretrained Models for Differential Equation Foundation Modelcited by 18
  34. 2024Characterizing Prompt Compression Methods for Long Context Inferencecited by 18
  35. 2024Fundamental Limits of Prompt Compression: A Rate-Distortion Framework for Black-Box Language Modelscited by 18
  36. 2024Salute the Classic: Revisiting Challenges of Machine Translation in the Age of Large Language Modelscited by 17
  37. 2025Sculptor: Empowering LLMs with Cognitive Agency via Active Context Managementcited by 17
  38. 2024Compressing Long Context for Enhancing RAG with AMR-based Concept Distillationcited by 16

50 most cited of 185 All 185 (JSON)