# SCOPE: A Generative Approach for LLM Prompt Compression

arXiv: https://arxiv.org/abs/2508.15813 · HTML: https://mlsanity.com/paper/2508.15813/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- Placed in Prompt and context compression, one of 12 directions in Efficient LLM inference and serving.
- Builds on 19 of its 27 references that are in the map.
- First posted on arXiv 16 Aug 2025.

## Placed in

- Efficient LLM inference and serving / [Prompt and context compression](https://mlsanity.com/inference/prompt-and-context-compression/)

## Builds on (most cited first, 19 of 19)

- [BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension](https://mlsanity.com/paper/1910.13461/) (2019)
- [BERTScore: Evaluating Text Generation with BERT](https://mlsanity.com/paper/1904.09675/) (2019)
- [TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehension](https://mlsanity.com/paper/1705.03551/) (2017)
- [LongBench: A Bilingual, Multitask Benchmark for Long Context Understanding](https://mlsanity.com/paper/2308.14508/) (2023)
- [PEGASUS: Pre-training with Extracted Gap-sentences for Abstractive Summarization](https://mlsanity.com/paper/1912.08777/) (2019)
- [A Discourse-Aware Attention Model for Abstractive Summarization of Long Documents](https://mlsanity.com/paper/1804.05685/) (2018)
- [LongLLMLingua: Accelerating and Enhancing LLMs in Long Context Scenarios via Prompt Compression](https://mlsanity.com/paper/2310.06839/) (2023)
- [LLMLingua-2: Data Distillation for Efficient and Faithful Task-Agnostic Prompt Compression](https://mlsanity.com/paper/2403.12968/) (2024)
- [Efficient Attentions for Long Document Summarization](https://mlsanity.com/paper/2104.02112/) (2021)
- [RECOMP: Improving Retrieval-Augmented LMs with Compression and Selective Augmentation](https://mlsanity.com/paper/2310.04408/) (2023)
- [Qwen2.5-1M Technical Report](https://mlsanity.com/paper/2501.15383/) (2025)
- [RetrievalAttention: Accelerating Long-Context LLM Inference via Vector Retrieval](https://mlsanity.com/paper/2409.10516/) (2024)
- [Jina Embeddings 2: 8192-Token General-Purpose Text Embeddings for Long Documents](https://mlsanity.com/paper/2310.19923/) (2023)
- [Unlocking Context Constraints of LLMs: Enhancing Context Efficiency of LLMs with Self-Information-Based Content Filtering](https://mlsanity.com/paper/2304.12102/) (2023)
- [CompAct: Compressing Retrieved Documents Actively for Question Answering](https://mlsanity.com/paper/2407.09014/) (2024)
- [PRCA: Fitting Black-Box Large Language Models for Retrieval Question Answering via Pluggable Reward-Driven Contextual Adapter](https://mlsanity.com/paper/2310.18347/) (2023)
- [Prompt-SAW: Leveraging Relation-Aware Graphs for Textual Prompt Compression](https://mlsanity.com/paper/2404.00489/) (2024)
- [Style-Compress: An LLM-Based Prompt Compression Framework Considering Task-Specific Styles](https://mlsanity.com/paper/2410.14042/) (2024)
- [Parse Trees Guided LLM Prompt Compression](https://mlsanity.com/paper/2409.15395/) (2024)

## Built on by (most cited first, 6 of 6)

- [Beyond Position Bias: Shifting Context Compression from Position-Driven to Semantic-Driven](https://mlsanity.com/paper/2605.09463/) (2026)
- [CompactPrompt: A Unified Pipeline for Prompt Data Compression in LLM Workflows](https://mlsanity.com/paper/2510.18043/) (2025)
- [When Summaries Distort Decisions: Information Fidelity in LLM-Compressed Financial Analysis](https://mlsanity.com/paper/2606.29251/) (2026)
- [Cmprsr: Abstractive Token-Level Question-Agnostic Prompt Compressor](https://mlsanity.com/paper/2511.12281/) (2025)
- [Text-Preserving Lossy Text Compression: A Study of Strategic Deletion and LLM Reconstruction](https://mlsanity.com/paper/2605.29000/) (2026)
- [Beyond Single-Modal Analytics: A Framework for Integrating Heterogeneous LLM-Based Query Systems for Multi-Modal Data](https://mlsanity.com/paper/2602.01701/) (2026)

## Abstract (arXiv)

A big issue in modern LLM applications is they tend to feed long context to LLM, which results in high inference cost and latency, and may exceed the context limit. Prompt compression addresses this issue by reducing the length of input context with minimum loss of generation quality, i.e, the goal of prompt compression is to shorten the LLM input while maintaining a high generation quality. To overcome these limitations, we propose SCOPE, a training-free generative prompt compression framework based on chunk-level rewriting. Unlike the existing token removal methods, our method centers at a chunking-and-summarization mechanism. Specifically, SCOPE splits a prompt into semantically coherent chunks and rewrites the chunks to be more concise. Then the chunks are reconstructed into a meaningful prompt. Additionally, we design several optimization techniques for SCOPE, effectively preserving critical information and text coherence in compression, as well as providing finer-grained control of the compression ratio. We conduct extensive evaluation on typical LLM applications like question-answering and summarization. Results show that SCOPE consistently outperforms the evaluated selective compression baselines across most settings, with particularly strong gains at high compression ratios.