mlsanityData to 14 Oct 2026

Task-agnostic Prompt Compression with Context-aware Sentence Embedding and Reward-guided Task Descriptor

arXiv 2502.13374 ↗First posted 19 Feb 2025cs.CLcited by 0

On the map: Efficient LLM inference and serving → Prompt and context compression

Abstract (arXiv)

The rise of Large Language Models (LLMs) has led to significant interest in prompt compression, a technique aimed at reducing the length of input prompts while preserving critical information. However, the prominent approaches in prompt compression often require explicit questions or handcrafted templates for compression, limiting their generalizability. We propose Task-agnostic Prompt Compression (TPC), a novel framework that generalizes compression across tasks and domains without requiring input questions or templates. TPC generates a context-relevant task description using a task descriptor trained on a curated dataset of context and query pairs, and fine-tuned via reinforcement learning with a reward function designed to capture the most relevant information. The task descriptor is then utilized to compute the relevance of each sentence in the prompt to generate the compressed prompt. We introduce 3 model sizes (Base, Large, and Huge), where the largest model outperforms the existing state-of-the-art methods on LongBench and ZeroSCROLLS benchmarks, and our smallest model performs comparable to the existing solutions while being considerably smaller.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Prompt and context compression, one of 12 directions in Efficient LLM inference and serving. Builds on 35 of its 42 references that are in the map. First posted on arXiv 19 Feb 2025.

Builds on

35 of 42 references.

Closest ancestors · 5

Same direction, newest first

  1. 2024Prompt Compression with Context-Aware Sentence Encoding for Fast and Improved LLM Inference2 Sep
  2. 2024Adapting LLMs for Efficient Context Processing through Soft Prompt Compression7 Apr
  3. 2024LLMLingua-2: Data Distillation for Efficient and Faithful Task-Agnostic Prompt Compression19 Mar
  4. 2023LongLLMLingua: Accelerating and Enhancing LLMs in Long Context Scenarios via Prompt Compression10 Oct
  5. 2023Discrete Prompt Compression with Reinforcement Learning17 Aug

Other references · 30

  1. 2017Attention Is All You Needcited by 60,361
  2. 2024The Llama 3 Herd of Modelscited by 14,715
  3. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,459
  4. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,297
  5. 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,170
  6. 2024Qwen2.5 Technical Reportcited by 4,942
  7. 2013Efficient Estimation of Word Representations in Vector Spacecited by 4,375
  8. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,780
  9. 2020Longformer: The Long-Document Transformercited by 3,413
  10. 2023Mistral 7Bcited by 3,164
  11. 2019The Curious Case of Neural Text Degenerationcited by 2,916
  12. 2022Self-Instruct: Aligning Language Models with Self-Generated Instructionscited by 2,724
18 more
  1. 2020The Pile: An 800GB Dataset of Diverse Text for Language Modelingcited by 2,411
  2. 2021Multitask Prompted Training Enables Zero-Shot Task Generalizationcited by 1,470
  3. 2023LongBench: A Bilingual, Multitask Benchmark for Long Context Understandingcited by 1,417
  4. 2022Text Embeddings by Weakly-Supervised Contrastive Pre-trainingcited by 1,253
  5. 2023Enhancing Chat Language Models by Scaling High-quality Instructional Conversationscited by 839
  6. 2023Towards General Text Embeddings with Multi-stage Contrastive Learningcited by 723
  7. 2024LLM2Vec: Large Language Models Are Secretly Powerful Text Encoderscited by 404
  8. 2023Orca: Progressive Learning from Complex Explanation Traces of GPT-4cited by 379
  9. 2023Adapting Language Models to Compress Contextscited by 356
  10. 2023LLMLingua: Compressing Prompts for Accelerated Inference of Large Language Modelscited by 287
  11. 2023Camels in a Changing Climate: Enhancing LM Adaptation with Tulu 2cited by 229
  12. 2022Recurrent Memory Transformercited by 203
  13. 2023Compressing Context to Enhance Inference Efficiency of Large Language Modelscited by 189
  14. 2023ZeroSCROLLS: A Zero-Shot Benchmark for Long Text Understandingcited by 176
  15. 2021Learned Token Pruning for Transformerscited by 139
  16. 2023Walking Down the Memory Maze: Beyond Context Limit through Interactive Readingcited by 133
  17. 2021Keep it Simple: Unsupervised Simplification of Multi-Paragraph Textcited by 45
  18. 2025A Closer Look at System Prompt Robustnesscited by 32

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025DAC: A Dynamic Attention-aware Approach for Task-Agnostic Prompt Compression
  2. 2024TACO-RL: Task Aware Prompt Compression Optimization with Reinforcement Learning
  3. 2026Lexical Prompt Compression for Large Language Models: A Training-Free, Deterministic Pipeline with Empirical Pareto Analysis Across Eleven Task Categories
  4. 2025Dynamic Compressing Prompts for Efficient Inference of Large Language Models
  5. 2024Prompt Compression for Large Language Models: A Survey
  6. 2024Style-Compress: An LLM-Based Prompt Compression Framework Considering Task-Specific Styles
  7. 2025Understanding and Improving Information Preservation in Prompt Compression for LLMs
  8. 2024Prompt-SAW: Leveraging Relation-Aware Graphs for Textual Prompt Compression