mlsanityData to 7 Oct 2026

Prompt Compression with Context-Aware Sentence Encoding for Fast and Improved LLM Inference

arXiv 2409.01227 ↗First posted 2 Sep 2024cs.CL cs.LGcited by 51

On the map: Efficient LLM inference and serving → Prompt and context compression

Abstract (arXiv)

Large language models (LLMs) have triggered a new stream of research focusing on compressing the context length to reduce the computational cost while ensuring the retention of helpful information for LLMs to answer the given question. Token-based removal methods are one of the most prominent approaches in this direction, but risk losing the semantics of the context caused by intermediate token removal, especially under high compression ratios, while also facing challenges in computational efficiency. In this work, we propose context-aware prompt compression (CPC), a sentence-level prompt compression technique where its key innovation is a novel context-aware sentence encoder that provides a relevance score for each sentence for a given question. To train this encoder, we generate a new dataset consisting of questions, positives, and negative pairs where positives are sentences relevant to the question, while negatives are irrelevant context sentences. We train the encoder in a contrastive setup to learn context-aware sentence representations. Our method considerably outperforms prior works on prompt compression on benchmark datasets and is up to 10.93x faster at inference compared to the best token-level compression method. We also find better improvement for shorter length constraints in most benchmarks, showing the effectiveness of our proposed solution in the compression of relevant information in a shorter context. Finally, we release the code and the dataset for quick reproducibility and further development: https://github.com/Workday/cpc.

Key facts. Cited by 51 papers in the map (data through 7 Oct 2026). Placed in Prompt and context compression, one of 12 directions in Efficient LLM inference and serving. Builds on 17 of its 29 references that are in the map. First posted on arXiv 2 Sep 2024.

Builds on

17 of 29 references.

Closest ancestors · 3

Same direction, newest first

  1. 2024Adapting LLMs for Efficient Context Processing through Soft Prompt Compression7 Apr
  2. 2024LLMLingua-2: Data Distillation for Efficient and Faithful Task-Agnostic Prompt Compression19 Mar
  3. 2023LongLLMLingua: Accelerating and Enhancing LLMs in Long Context Scenarios via Prompt Compression10 Oct

Other references · 14

  1. 2017Attention Is All You Needcited by 59,932
  2. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  3. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,160
  4. 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,049
  5. 2020Longformer: The Long-Document Transformercited by 3,362
  6. 2016Pointer Sentinel Mixture Modelscited by 2,992
  7. 2022A Survey on In-context Learningcited by 717
  8. 2023Towards General Text Embeddings with Multi-stage Contrastive Learningcited by 705
  9. 2024LLM2Vec: Large Language Models Are Secretly Powerful Text Encoderscited by 396
  10. 2023Adapting Language Models to Compress Contextscited by 350
  11. 2023ZeroSCROLLS: A Zero-Shot Benchmark for Long Text Understandingcited by 175
  12. 2023Walking Down the Memory Maze: Beyond Context Limit through Interactive Readingcited by 129
2 more
  1. 2021Keep it Simple: Unsupervised Simplification of Multi-Paragraph Textcited by 45
  2. 2022Efficient Unsupervised Sentence Compression by Fine-tuning Transformers with Reinforcement Learningcited by 18

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 2; 2025: 24; 2026: 25.

51 papers.

Sort:

By area:

  1. 2025LightMem: Lightweight and Efficient Memory-Augmented Generationcited by 163
  2. 2024Prompt Compression for Large Language Models: A Surveycited by 65
  3. 2025Reducing Cost of LLM Agents with Trajectory Reductioncited by 44
  4. 2025Vibe Coding vs. Agentic Coding: Fundamentals and Practical Implications of Agentic AIcited by 39
  5. 2025A Survey of Multimodal Retrieval-Augmented Generationcited by 37
  6. 2025Context Engineering 2.0: The Context of Context Engineeringcited by 17
  7. 2026COMI: Coarse-to-fine Context Compression via Marginal Information Gaincited by 13
  8. 2025Efficient Prompt Compression with Evaluator Heads for Long-Context Transformer Inferencecited by 10
  9. 2026Read As Human: Compressing Context via Parallelizable Close Reading and Skimmingcited by 10
  10. 2025DSPC: Dual-Stage Progressive Compression Framework for Efficient Long-Context Reasoningcited by 9
  11. 2025The Power of Personality: A Human Simulation Perspective to Investigate Large Language Model Agentscited by 8
  12. 2025REFRAG: Rethinking RAG based Decodingcited by 8
38 more
  1. 2025AdmTree: Compressing Lengthy Context with Adaptive Semantic Treescited by 7
  2. 2025What Makes a Good Natural Language Prompt?cited by 6
  3. 2025Draft-based Approximate Inference for LLMscited by 6
  4. 2025CompLLM: Compression for Long Context Q&Acited by 5
  5. 2026From History to State: Constant-Context Skill Learning for LLM Agentscited by 5
  6. 2025Key, Value, Compress: A Systematic Exploration of KV Cache Compression Techniquescited by 4
  7. 2025MOOSComp: Improving Lightweight Long-Context Compressor via Mitigating Over-Smoothing and Incorporating Outlier Scorescited by 4
  8. 2025Sentinel: Decoding Context Utilization via Attention Probing for Efficient LLM Context Compressioncited by 4
  9. 2025Lossless Token Sequence Compression via Meta-Tokenscited by 4
  10. 2026CoACT: Action-Preserving Observation Compression for Coding Agentscited by 4
  11. 2024TRIM: Token Reduction and Inference Modeling for Cost-Effective Language Generationcited by 2
  12. 2026Tackling the Inherent Difficulty of Noise Filtering in RAGcited by 2
  13. 2026BEAVER: A Training-Free Hierarchical Prompt Compression Method via Structure-Aware Page Selectioncited by 2
  14. 2026From Token to Token Pair: Efficient Prompt Compression for Large Language Models in Clinical Predictioncited by 2
  15. 2025CCF: A Context Compression Framework for Efficient Long-Sequence Language Modelingcited by 1
  16. 2025HarmTransform: Transforming Explicit Harmful Queries into Stealthy via Multi-Agent Debatecited by 1
  17. 2026From Similarity to Structure: Training-free LLM Context Compression with Hybrid Graph Priorscited by 1
  18. 2026IterCOMP: Reasoning-aware Adaptive Prompt Compression for Multi-hop Question Answeringcited by 1
  19. 2026Lost in Compression: A Controlled Cross-Lingual Audit of Extractive Prompt Compressorscited by 1
  20. 2026AttnCompress: Dynamic Attention-Guided Trajectory Compression for Software Engineering Agentscited by 1
  21. 2025Task-agnostic Prompt Compression with Context-aware Sentence Embedding and Reward-guided Task Descriptorcited by 0
  22. 2025LTG at SemEval-2025 Task 10: Optimizing Context for Classification of Narrative Rolescited by 0
  23. 2025Mockingbird: How does LLM perform in general machine learning tasks?cited by 0
  24. 2025Rethinking Inference Placement for Deep Learning across Edge and Cloud Platforms: A Multi-Objective Optimization Perspective and Future Directionscited by 0
  25. 2025Cmprsr: Abstractive Token-Level Question-Agnostic Prompt Compressorcited by 0
  26. 2026UT-ACA: Uncertainty-Triggered Adaptive Context Allocation for Long-Context Inferencecited by 0
  27. 2026Compressing Sequences in the Latent Embedding Space: K-Token Merging for Large Language Modelscited by 0
  28. 2026ONTO: A Token-Efficient Columnar Notation for LLM Input Optimizationcited by 0
  29. 2026An Approximation Algorithm for Graph Label Selectioncited by 0
  30. 2026EASE-TTT: Evidence-Aligned Selective Test-Time Training for Long-Context Question Answeringcited by 0
  31. 2026Mapping Text to Multiplex Graph: Prompt Compression as L'evy Walk-Guided Graph Pruningcited by 0
  32. 2026SALT: Salience-Aware Lexical Trie for Long-Context Compressioncited by 0
  33. 2026Every Time I Hire a Linguist, Inference Costs Go Down: On Linguistic Rules as Effective Prompt Compressorscited by 0
  34. 2026Leveraging Trajectory Graphs for Pre-Execution Error Diagnosis in Agentic LLM Systemscited by 0
  35. 2026Relevant but Incomplete: Referential Dangling as a Paradigm-Level Failure Mode in Hard Prompt Compressioncited by 0
  36. 2026An Exploratory Evaluation of LLM-Assisted Rewriting of Moderate-Complexity Financial Sentences for DisCoCat-Based Sentiment Analysiscited by 0
  37. 2026Compression Beyond the Uncompressed: A Two-Stage Training Recipe for Soft Context Compression in RAGcited by 0
  38. 2026PC-SubMax: Efficient Prompt Compression via Regularized Submodular Maximizationcited by 0

50 most cited of 51 All 51 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2026EnComp: Lightweight Encoder-Only Context Compression for Retrieval-Augmented Question Answering
  2. 2024Recurrent Context Compression: Efficiently Expanding the Context Window of LLM
  3. 2024500xCompressor: Generalized Prompt Compression for Large Language Models
  4. 2026Compressing Long Context into Answer-Aligned Memory Embeddings for LLM Inference
  5. 2026Lexical Prompt Compression for Large Language Models: A Training-Free, Deterministic Pipeline with Empirical Pareto Analysis Across Eleven Task Categories
  6. 2025SCOPE: A Generative Approach for LLM Prompt Compression
  7. 2024Characterizing Prompt Compression Methods for Long Context Inference
  8. 2024Retaining Key Information under High Compression Ratios: Query-Guided Compressor for LLMs