mlsanityData to 7 Oct 2026

LongLLMLingua: Accelerating and Enhancing LLMs in Long Context Scenarios via Prompt Compression

arXiv 2310.06839 ↗First posted 10 Oct 2023cs.CL cs.LGcited by 443

On the map: Efficient LLM inference and serving → Prompt and context compression

Abstract (arXiv)

In long context scenarios, large language models (LLMs) face three main challenges: higher computational cost, performance reduction, and position bias. Research indicates that LLM performance hinges on the density and position of key information in the input prompt. Inspired by these findings, we propose LongLLMLingua for prompt compression towards improving LLMs' perception of the key information to simultaneously address the three challenges. Our extensive evaluation across various long context scenarios demonstrates that LongLLMLingua not only enhances performance but also significantly reduces costs and latency. For instance, in the NaturalQuestions benchmark, LongLLMLingua boosts performance by up to 21.4% with around 4x fewer tokens in GPT-3.5-Turbo, leading to substantial cost savings. It achieves a 94.0% cost reduction in the LooGLE benchmark. Moreover, when compressing prompts of about 10k tokens at ratios of 2x-6x, LongLLMLingua can accelerate end-to-end latency by 1.4x-2.6x. Our code is available at https://aka.ms/LongLLMLingua.

Key facts. Cited by 443 papers in the map (data through 7 Oct 2026). Placed in Prompt and context compression, one of 12 directions in Efficient LLM inference and serving. Builds on 27 of its 49 references that are in the map. First posted on arXiv 10 Oct 2023.

Builds on

27 of 49 references.

Closest ancestors · 1

Same direction, newest first

  1. 2023In-context Autoencoder for Context Compression in a Large Language Model13 Jul

Other references · 26

  1. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,160
  2. 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,049
  3. 2023Generative Agents: Interactive Simulacra of Human Behaviorcited by 3,500
  4. 2023Lost in the Middle: How Language Models Use Long Contextscited by 3,383
  5. 2023LongBench: A Bilingual, Multitask Benchmark for Long Context Understandingcited by 1,378
  6. 2021Unsupervised Dense Information Retrieval with Contrastive Learningcited by 1,203
  7. 2021Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolationcited by 1,085
  8. 2022Token Merging: Your ViT But Fastercited by 995
  9. 2023Large Language Models Can Be Easily Distracted by Irrelevant Contextcited by 820
  10. 2023Extending Context Window of Large Language Models via Positional Interpolationcited by 721
  11. 2023Retentive Network: A Successor to Transformer for Large Language Modelscited by 594
  12. 2023YaRN: Efficient Context Window Extension of Large Language Modelscited by 584
14 more
  1. 2021MetaICL: Learning to Learn In Contextcited by 450
  2. 2023Learning to Compress Prompts with Gist Tokenscited by 379
  3. 2023Adapting Language Models to Compress Contextscited by 350
  4. 2023Effective Long-Context Scaling of Foundation Modelscited by 301
  5. 2023LLMLingua: Compressing Prompts for Accelerated Inference of Large Language Modelscited by 279
  6. 2023LongNet: Scaling Transformers to 1,000,000,000 Tokenscited by 201
  7. 2023ZeroSCROLLS: A Zero-Shot Benchmark for Long Text Understandingcited by 175
  8. 2022Self-Adaptive In-Context Learning: An Information Compression Perspective for In-Context Example Selection and Orderingcited by 158
  9. 2023Focused Transformer: Contrastive Training for Context Scalingcited by 133
  10. 2023Unlimiformer: Long-Range Transformers with Unlimited Length Inputcited by 127
  11. 2020Length-Adaptive Transformer: Train Once with Length Drop, Use Anytime with Searchcited by 76
  12. 2023Unlocking Context Constraints of LLMs: Enhancing Context Efficiency of LLMs with Self-Information-Based Content Filteringcited by 62
  13. 2022AdapLeR: Speeding up Inference by Adaptive Length Reductioncited by 19
  14. 2023XGen-7B Technical Reportcited by 11

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 9; 2024: 106; 2025: 136; 2026: 192.

443 papers.

Sort:

By area:

  1. 2023Retrieval-Augmented Generation for Large Language Models: A Surveycited by 2,185
  2. 2024RULER: What's the Real Context Size of Your Long-Context Language Models?cited by 1,270
  3. 2024MInference 1.0: Accelerating Pre-filling for Long-Context LLMs via Dynamic Sparse Attentioncited by 375
  4. 2024LLMLingua-2: Data Distillation for Efficient and Faithful Task-Agnostic Prompt Compressioncited by 343
  5. 2025R1-Searcher: Incentivizing the Search Capability in LLMs via Reinforcement Learningcited by 322
  6. 2024Chain of Agents: Large Language Models Collaborating on Long-Context Taskscited by 201
  7. 2023CacheGen: KV Cache Compression and Streaming for Fast Large Language Model Servingcited by 199
  8. 2024A Survey on Efficient Inference for Large Language Modelscited by 197
  9. 2023Efficient Large Language Models: A Surveycited by 183
  10. 2024FlashRAG: A Modular Toolkit for Efficient Retrieval-Augmented Generation Researchcited by 165
  11. 2025LightMem: Lightweight and Efficient Memory-Augmented Generationcited by 163
  12. 2024LLM Maybe LongLM: Self-Extend LLM Context Window Without Tuningcited by 133
38 more
  1. 2024xRAG: Extreme Context Compression for Retrieval-augmented Generation with One Tokencited by 123
  2. 2025FlexPrefill: A Context-Aware Sparse Attention Mechanism for Efficient Long-Sequence Inferencecited by 121
  3. 2024MemoRAG: Boosting Long Context Processing with Global Memory-Enhanced Retrieval Augmentationcited by 118
  4. 2024C3oT: Generating Shorter Chain-of-Thought without Compromising Effectivenesscited by 117
  5. 2024LongAlign: A Recipe for Long Context Alignment of Large Language Modelscited by 102
  6. 2023Towards Efficient Generative Large Language Model Serving: A Survey from Algorithms to Systemscited by 101
  7. 2024Long Context Compression with Activation Beaconcited by 101
  8. 2025On Memory Construction and Retrieval for Personalized Conversational Agentscited by 101
  9. 2024Found in the Middle: Calibrating Positional Attention Bias Improves Long Context Utilizationcited by 97
  10. 2024Keep the Cost Down: A Review on Methods to Optimize LLM' s KV-Cache Consumptioncited by 96
  11. 2024Not All Heads Matter: A Head-Level KV Cache Compression Method with Integrated Retrieval and Reasoningcited by 96
  12. 2024From Matching to Generation: A Survey on Generative Information Retrievalcited by 89
  13. 2024Searching for Best Practices in Retrieval-Augmented Generationcited by 83
  14. 2025Lifelong Learning of Large Language Model based Agents: A Roadmapcited by 78
  15. 2024Found in the Middle: How Language Models Use Long Contexts Better via Plug-and-Play Positional Encodingcited by 72
  16. 2023Thread of Thought Unraveling Chaotic Contextscited by 71
  17. 2024Beyond the Limits: A Survey of Techniques to Extend the Context Length in Large Language Modelscited by 71
  18. 2025Process vs. Outcome Reward: Which is Better for Agentic RAG Reinforcement Learningcited by 68
  19. 2024Retrieval Augmented Generation (RAG) and Beyond: A Comprehensive Survey on How to Make your LLMs use External Data More Wiselycited by 65
  20. 2024Prompt Compression for Large Language Models: A Surveycited by 65
  21. 2024CompAct: Compressing Retrieved Documents Actively for Question Answeringcited by 64
  22. 2024Layer-Condensed KV Cache for Efficient Inference of Large Language Modelscited by 58
  23. 2025ChunkKV: Semantic-Preserving KV Cache Compression for Efficient Long-Context LLM Inferencecited by 57
  24. 2024Prompt Compression with Context-Aware Sentence Encoding for Fast and Improved LLM Inferencecited by 51
  25. 2024DocETL: Agentic Query Rewriting and Evaluation for Complex Document Processingcited by 49
  26. 2024LongRAG: A Dual-Perspective Retrieval-Augmented Generation Paradigm for Long-Context Question Answeringcited by 49
  27. 2024Efficient Prompting Methods for Large Language Models: A Surveycited by 46
  28. 2025A Survey of Large Language Model Agents for Question Answeringcited by 46
  29. 2024DeciMamba: Exploring the Length Extrapolation Potential of Mambacited by 45
  30. 2024500xCompressor: Generalized Prompt Compression for Large Language Modelscited by 45
  31. 2024Modular RAG: Transforming RAG Systems into LEGO-like Reconfigurable Frameworkscited by 41
  32. 2024UDA: A Benchmark Suite for Retrieval Augmented Generation in Real-world Document Analysiscited by 40
  33. 2026SWE-Pruner: Self-Adaptive Context Pruning for Coding Agentscited by 40
  34. 2024Discovering the Gems in Early Layers: Accelerating Long-Context LLMs with 1000x Input Token Reductioncited by 39
  35. 2025LongCodeZip: Compress Long Context for Code Language Modelscited by 38
  36. 2024HtmlRAG: HTML is Better Than Plain Text for Modeling Retrieved Knowledge in RAG Systemscited by 37
  37. 2025A Survey of Multimodal Retrieval-Augmented Generationcited by 37
  38. 2025Cache-Craft: Managing Chunk-Caches for Efficient Retrieval-Augmented Generationcited by 36

50 most cited of 443 All 443 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2023Compressing LLMs: The Truth is Rarely Pure and Never Simple
  2. 2024KV Cache Compression, But What Must We Give in Return? A Comprehensive Benchmark of Long Context Capable Approaches
  3. 2025Sentence-Anchored Gist Compression for Long-Context LLMs
  4. 2024LLMC: Benchmarking Large Language Model Quantization with a Versatile Compression Toolkit
  5. 2025AccLLM: Accelerating Long-Context LLM Inference Via Algorithm-Hardware Co-Design
  6. 2025Long-Short Alignment for Effective Long-Context Modeling in LLMs
  7. 2024Enhancing and Accelerating Large Language Models via Instruction-Aware Contextual Compression
  8. 2024FocusLLM: Precise Understanding of Long Context by Dynamic Condensing