mlsanityData to 7 Oct 2026

Characterizing Prompt Compression Methods for Long Context Inference

arXiv 2407.08892 ↗First posted 11 Jul 2024cs.CL cs.LGcited by 18

On the map: Efficient LLM inference and serving → Prompt and context compression

Abstract (arXiv)

Long context inference presents challenges at the system level with increased compute and memory requirements, as well as from an accuracy perspective in being able to reason over long contexts. Recently, several methods have been proposed to compress the prompt to reduce the context length. However, there has been little work on comparing the different proposed methods across different tasks through a standardized analysis. This has led to conflicting results. To address this, here we perform a comprehensive characterization and evaluation of different prompt compression methods. In particular, we analyze extractive compression, summarization-based abstractive compression, and token pruning methods. Surprisingly, we find that extractive compression often outperforms all the other approaches, and enables up to 10x compression with minimal accuracy degradation. Interestingly, we also find that despite several recent claims, token pruning methods often lag behind extractive compression. We only found marginal improvements on summarization tasks.

Key facts. Cited by 18 papers in the map (data through 7 Oct 2026). Placed in Prompt and context compression, one of 12 directions in Efficient LLM inference and serving. Builds on 39 of its 54 references that are in the map. First posted on arXiv 11 Jul 2024.

Builds on

39 of 54 references.

Closest ancestors · 4

Same direction, newest first

  1. 2024Prompt-SAW: Leveraging Relation-Aware Graphs for Textual Prompt Compression30 Mar
  2. 2023LongLLMLingua: Accelerating and Enhancing LLMs in Long Context Scenarios via Prompt Compression10 Oct
  3. 2023Discrete Prompt Compression with Reinforcement Learning17 Aug
  4. 2023In-context Autoencoder for Context Compression in a Large Language Model13 Jul

Other references · 35

  1. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  2. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  3. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,160
  4. 2018HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answeringcited by 3,686
  5. 2023Lost in the Middle: How Language Models Use Long Contextscited by 3,383
  6. 2024Gemini 1.5: Unlocking multimodal understanding across millions of tokens of contextcited by 3,175
  7. 2020DeBERTa: Decoding-enhanced BERT with Disentangled Attentioncited by 2,157
  8. 2024Mixtral of Expertscited by 1,785
  9. 2023LongBench: A Bilingual, Multitask Benchmark for Long Context Understandingcited by 1,378
  10. 2023MemGPT: Towards LLMs as Operating Systemscited by 1,295
  11. 2020Constructing A Multi-hop QA Dataset for Comprehensive Evaluation of Reasoning Stepscited by 1,259
  12. 2022Text Embeddings by Weakly-Supervised Contrastive Pre-trainingcited by 1,216
23 more
  1. 2021Unsupervised Dense Information Retrieval with Contrastive Learningcited by 1,203
  2. 2023BloombergGPT: A Large Language Model for Financecited by 923
  3. 2024SnapKV: LLM Knows What You are Looking for Before Generationcited by 848
  4. 2023H2O: Heavy-Hitter Oracle for Efficient Generative Inference of Large Language Modelscited by 805
  5. 2019Passage Re-ranking with BERTcited by 629
  6. 2024KIVI: A Tuning-Free Asymmetric 2bit Quantization for KV Cachecited by 604
  7. 2017The NarrativeQA Reading Comprehension Challengecited by 590
  8. 2024KVQuant: Towards 10 Million Context Length LLM Inference with KV Cache Quantizationcited by 571
  9. 2021A Dataset of Information-Seeking Questions and Answers Anchored in Research Paperscited by 475
  10. 2023Model Tells You What to Discard: Adaptive KV Cache Compression for LLMscited by 466
  11. 2019Multi-News: a Large-Scale Multi-Document Summarization Dataset and Abstractive Hierarchical Modelcited by 431
  12. 2021QMSum: A New Benchmark for Query-based Multi-domain Meeting Summarizationcited by 363
  13. 2023Adapting Language Models to Compress Contextscited by 350
  14. 2021Efficient Attentions for Long Document Summarizationcited by 342
  15. 2023RECOMP: Improving Retrieval-Augmented LMs with Compression and Selective Augmentationcited by 253
  16. 2023FinGPT: Open-Source Financial Large Language Modelscited by 244
  17. 2020SpAtten: Efficient Sparse Attention Architecture with Cascade Token and Head Pruningcited by 228
  18. 2023Compressing Context to Enhance Inference Efficiency of Large Language Modelscited by 185
  19. 2023RankZephyr: Effective and Robust Zero-Shot Listwise Reranking is a Breeze!cited by 151
  20. 2021Learned Token Pruning for Transformerscited by 138
  21. 2020Length-Adaptive Transformer: Train Once with Length Drop, Use Anytime with Searchcited by 76
  22. 2019Extractive Summarization of Long Documents by Combining Global and Local Contextcited by 59
  23. 2023PRCA: Fitting Black-Box Large Language Models for Retrieval Question Answering via Pluggable Reward-Driven Contextual Adaptercited by 38

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 1; 2025: 8; 2026: 9.

18 papers.

Sort:

By area:

  1. 2024Efficient and Scalable Estimation of Tool Representations in Vector Spacecited by 22
  2. 2025Efficient Prompt Compression with Evaluator Heads for Long-Context Transformer Inferencecited by 10
  3. 2025Text or Pixels? It Takes Half: On the Token Efficiency of Visual Text Inputs in Multimodal LLMscited by 8
  4. 2026Prompt Compression in the Wild: Measuring Latency, Rate Adherence, and Quality for Faster LLM Inferencecited by 8
  5. 2025Beyond Hard and Soft: Hybrid Context Compression for Balancing Local and Global Information Retentioncited by 7
  6. 2025Beyond RAG: Task-Aware KV Cache Compression for Comprehensive Knowledge Reasoningcited by 4
  7. 2025PIS: Linking Importance Sampling and Attention Mechanisms for Efficient Prompt Compressioncited by 4
  8. 2026Rethinking Soft Compression in Retrieval-Augmented Generation: A Query-Conditioned Selector Perspectivecited by 4
  9. 2025SALE : Low-bit Estimation for Efficient Sparse Attention in Long-context LLM Prefillingcited by 3
  10. 2026Compression Method Matters: Benchmark-Dependent Output Dynamics in LLM Prompt Compressioncited by 2
  11. 2025ACoRN: Noise-Robust Abstractive Compression in Retrieval-Augmented Language Modelscited by 1
  12. 2026Reliability-Aware Determinantal Point Processes for Robust Informative Data Selection in Large Language Modelscited by 1
6 more
  1. 2026Fixed RAG Compression Collapses Measured Reader Scalingcited by 1
  2. 2025Noise-Robust Abstractive Compression in Retrieval-Augmented Language Modelscited by 0
  3. 2026Mitigating Batch Effects in Histopathology via Language-Mediated Robust Embedding Generationcited by 0
  4. 2026SALT: Salience-Aware Lexical Trie for Long-Context Compressioncited by 0
  5. 2026Learning What to Remember: Test-Time Training via Context Distillationcited by 0
  6. 2026Relevant but Incomplete: Referential Dangling as a Paradigm-Level Failure Mode in Hard Prompt Compressioncited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Prompt Compression with Context-Aware Sentence Encoding for Fast and Improved LLM Inference
  2. 2025No Mean Feat: Simple, Strong Baselines for Context Compression
  3. 2026LongAttnComp: Cross-Family Context Compression for Long-Context Reasoning
  4. 2025Understanding and Improving Information Preservation in Prompt Compression for LLMs
  5. 2026Lexical Prompt Compression for Large Language Models: A Training-Free, Deterministic Pipeline with Empirical Pareto Analysis Across Eleven Task Categories
  6. 2026PReM: Learning What to Preserve and When to Refresh for Context Compression
  7. 2024Recurrent Context Compression: Efficiently Expanding the Context Window of LLM
  8. 2026Cognitive Chunking for Soft Prompts: Accelerating Compressor Learning via Block-wise Causal Masking