# Prompt Compression for Large Language Models: A Survey

arXiv: https://arxiv.org/abs/2410.12388 · HTML: https://mlsanity.com/paper/2410.12388/

## Key facts

- Cited by 67 papers in the map (data through 14 Oct 2026).
- Placed in Prompt and context compression, one of 12 directions in Efficient LLM inference and serving.
- Builds on 42 of its 58 references that are in the map.
- First posted on arXiv 16 Oct 2024.

## Placed in

- Efficient LLM inference and serving / [Prompt and context compression](https://mlsanity.com/inference/prompt-and-context-compression/)

## Builds on (most cited first, 30 of 42)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [The Llama 3 Herd of Models](https://mlsanity.com/paper/2407.21783/) (2024)
- [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://mlsanity.com/paper/2307.09288/) (2023)
- [Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks](https://mlsanity.com/paper/2005.11401/) (2020)
- [SQuAD: 100,000+ Questions for Machine Comprehension of Text](https://mlsanity.com/paper/1606.05250/) (2016)
- [Prefix-Tuning: Optimizing Continuous Prompts for Generation](https://mlsanity.com/paper/2101.00190/) (2021)
- [The Power of Scale for Parameter-Efficient Prompt Tuning](https://mlsanity.com/paper/2104.08691/) (2021)
- [HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answering](https://mlsanity.com/paper/1809.09600/) (2018)
- [QLoRA: Efficient Finetuning of Quantized LLMs](https://mlsanity.com/paper/2305.14314/) (2023)
- [Longformer: The Long-Document Transformer](https://mlsanity.com/paper/2004.05150/) (2020)
- [Neural Machine Translation of Rare Words with Subword Units](https://mlsanity.com/paper/1508.07909/) (2015)
- [The Pile: An 800GB Dataset of Diverse Text for Language Modeling](https://mlsanity.com/paper/2101.00027/) (2020)
- [Don't Stop Pretraining: Adapt Language Models to Domains and Tasks](https://mlsanity.com/paper/2004.10964/) (2020)
- [Generating Long Sequences with Sparse Transformers](https://mlsanity.com/paper/1904.10509/) (2019)
- [RACE: Large-scale ReAding Comprehension Dataset From Examinations](https://mlsanity.com/paper/1704.04683/) (2017)
- [Vision-Language Models for Vision Tasks: A Survey](https://mlsanity.com/paper/2304.00685/) (2023)
- [DoRA: Weight-Decomposed Low-Rank Adaptation](https://mlsanity.com/paper/2402.09353/) (2024)
- [Prompt Programming for Large Language Models: Beyond the Few-Shot Paradigm](https://mlsanity.com/paper/2102.07350/) (2021)
- [LongLLMLingua: Accelerating and Enhancing LLMs in Long Context Scenarios via Prompt Compression](https://mlsanity.com/paper/2310.06839/) (2023)
- [Learning to Compress Prompts with Gist Tokens](https://mlsanity.com/paper/2304.08467/) (2023)
- [Adapting Language Models to Compress Contexts](https://mlsanity.com/paper/2305.14788/) (2023)
- [LLMLingua-2: Data Distillation for Efficient and Faithful Task-Agnostic Prompt Compression](https://mlsanity.com/paper/2403.12968/) (2024)
- [Reasoning with Language Model Prompting: A Survey](https://mlsanity.com/paper/2212.09597/) (2022)
- [xRAG: Extreme Context Compression for Retrieval-augmented Generation with One Token](https://mlsanity.com/paper/2405.13792/) (2024)
- [Prompt Compression and Contrastive Conditioning for Controllability and Toxicity Reduction in Language Models](https://mlsanity.com/paper/2210.03162/) (2022)
- [Efficient Multimodal Large Language Models: A Survey](https://mlsanity.com/paper/2405.10739/) (2024)
- [Combiner: Full Attention Transformer with Sparse Computation Cost](https://mlsanity.com/paper/2107.05768/) (2021)
- [CompAct: Compressing Retrieved Documents Actively for Question Answering](https://mlsanity.com/paper/2407.09014/) (2024)
- [Learning to Compress Prompt in Natural Language Formats](https://mlsanity.com/paper/2402.18700/) (2024)

## Built on by (most cited first, 30 of 67)

- [LightMem: Lightweight and Efficient Memory-Augmented Generation](https://mlsanity.com/paper/2510.18866/) (2025)
- [A Survey on Large Language Model Acceleration based on KV Cache Management](https://mlsanity.com/paper/2412.19442/) (2024)
- [Scaling LLM Multi-turn RL with End-to-end Summarization-based Context Management](https://mlsanity.com/paper/2510.06727/) (2025)
- [MAIN-RAG: Multi-Agent Filtering Retrieval-Augmented Generation](https://mlsanity.com/paper/2501.00332/) (2024)
- [LongCodeZip: Compress Long Context for Code Language Models](https://mlsanity.com/paper/2510.00446/) (2025)
- [Rethinking Memory in LLM based Agents: Representations, Operations, and Emerging Topics](https://mlsanity.com/paper/2505.00675/) (2025)
- [SkillReducer: Optimizing LLM Agent Skills for Token Efficiency](https://mlsanity.com/paper/2603.29919/) (2026)
- [A Survey of Token Compression for Efficient Multimodal Large Language Models](https://mlsanity.com/paper/2507.20198/) (2025)
- [Hierarchical Document Refinement for Long-context Retrieval-augmented Generation](https://mlsanity.com/paper/2505.10413/) (2025)
- [Autoencoding-Free Context Compression for LLMs via Contextual Semantic Anchors](https://mlsanity.com/paper/2510.08907/) (2025)
- [LightTransfer: Your Long-Context LLM is Secretly a Hybrid Model with Effortless Adaptation](https://mlsanity.com/paper/2410.13846/) (2024)
- [R3Mem: Bridging Memory Retention and Retrieval via Reversible Compression](https://mlsanity.com/paper/2502.15957/) (2025)
- [Enhancing RAG Efficiency with Adaptive Context Compression](https://mlsanity.com/paper/2507.22931/) (2025)
- [Beyond Hard and Soft: Hybrid Context Compression for Balancing Local and Global Information Retention](https://mlsanity.com/paper/2505.15774/) (2025)
- [A Survey on Prompt Tuning](https://mlsanity.com/paper/2507.06085/) (2025)
- [ReasonGraph: Visualisation of Reasoning Paths](https://mlsanity.com/paper/2503.03979/) (2025)
- [PT-MoE: An Efficient Finetuning Framework for Integrating Mixture-of-Experts into Prompt Tuning](https://mlsanity.com/paper/2505.09519/) (2025)
- [Memory Tokens: Large Language Models Can Generate Reversible Sentence Embeddings](https://mlsanity.com/paper/2506.15001/) (2025)
- [Semantic Caching of Contextual Summaries for Efficient Question-Answering with Language Models](https://mlsanity.com/paper/2505.11271/) (2025)
- [Agentic Systems in Radiology: Design, Applications, Evaluation, and Challenges](https://mlsanity.com/paper/2510.09404/) (2025)
- [Compositional Steering of Large Language Models with Steering Tokens](https://mlsanity.com/paper/2601.05062/) (2026)
- [MOOSComp: Improving Lightweight Long-Context Compressor via Mitigating Over-Smoothing and Incorporating Outlier Scores](https://mlsanity.com/paper/2504.16786/) (2025)
- [Lossless Token Sequence Compression via Meta-Tokens](https://mlsanity.com/paper/2506.00307/) (2025)
- [Scaling Up Efficient Small Language Models Serving and Deployment for Semantic Job Search](https://mlsanity.com/paper/2510.22101/) (2025)
- [ShortCoder: Knowledge-Augmented Syntax Optimization for Token-Efficient Code Generation](https://mlsanity.com/paper/2601.09703/) (2026)
- [A Guide to Large Language Models in Modeling and Simulation: From Core Techniques to Critical Challenges](https://mlsanity.com/paper/2602.05883/) (2026)
- [Beyond Position Bias: Shifting Context Compression from Position-Driven to Semantic-Driven](https://mlsanity.com/paper/2605.09463/) (2026)
- [Compress the Context, Keep the Commitments: A Formal Framework for Verifiable LLM Context Compression](https://mlsanity.com/paper/2605.17304/) (2026)
- [Mem-π: Adaptive Memory through Learning When and What to Generate](https://mlsanity.com/paper/2605.21463/) (2026)
- [ECHO: Prune To Act, Trace To Learn With Selective Turn Memory In Agentic RL](https://mlsanity.com/paper/2606.31650/) (2026)

## Abstract (arXiv)

Leveraging large language models (LLMs) for complex natural language tasks typically requires long-form prompts to convey detailed requirements and information, which results in increased memory usage and inference costs. To mitigate these challenges, multiple efficient methods have been proposed, with prompt compression gaining significant research interest. This survey provides an overview of prompt compression techniques, categorized into hard prompt methods and soft prompt methods. First, the technical approaches of these methods are compared, followed by an exploration of various ways to understand their mechanisms, including the perspectives of attention optimization, Parameter-Efficient Fine-Tuning (PEFT), modality integration, and new synthetic language. We also examine the downstream adaptations of various prompt compression techniques. Finally, the limitations of current prompt compression methods are analyzed, and several future directions are outlined, such as optimizing the compression encoder, combining hard and soft prompts methods, and leveraging insights from multimodality.