mlsanityData to 14 Oct 2026

Prompt Compression for Large Language Models: A Survey

arXiv 2410.12388 ↗First posted 16 Oct 2024cs.CLcited by 67

On the map: Efficient LLM inference and serving → Prompt and context compression

Abstract (arXiv)

Leveraging large language models (LLMs) for complex natural language tasks typically requires long-form prompts to convey detailed requirements and information, which results in increased memory usage and inference costs. To mitigate these challenges, multiple efficient methods have been proposed, with prompt compression gaining significant research interest. This survey provides an overview of prompt compression techniques, categorized into hard prompt methods and soft prompt methods. First, the technical approaches of these methods are compared, followed by an exploration of various ways to understand their mechanisms, including the perspectives of attention optimization, Parameter-Efficient Fine-Tuning (PEFT), modality integration, and new synthetic language. We also examine the downstream adaptations of various prompt compression techniques. Finally, the limitations of current prompt compression methods are analyzed, and several future directions are outlined, such as optimizing the compression encoder, combining hard and soft prompts methods, and leveraging insights from multimodality.

Key facts. Cited by 67 papers in the map (data through 14 Oct 2026). Placed in Prompt and context compression, one of 12 directions in Efficient LLM inference and serving. Builds on 42 of its 58 references that are in the map. First posted on arXiv 16 Oct 2024.

Builds on

42 of 58 references.

Closest ancestors · 11

Same direction, newest first

  1. 2024TACO-RL: Task Aware Prompt Compression Optimization with Reinforcement Learning19 Sep
  2. 2024Prompt Compression with Context-Aware Sentence Encoding for Fast and Improved LLM Inference2 Sep
  3. 2024Context Embeddings for Efficient Answer Generation in RAG12 Jul
  4. 2024Concise and Precise Context Compression for Tool-Using Language Models2 Jul
  5. 2024Retaining Key Information under High Compression Ratios: Query-Guided Compressor for LLMs4 Jun
  6. 2024xRAG: Extreme Context Compression for Retrieval-augmented Generation with One Token22 May
  7. 2024LLMLingua-2: Data Distillation for Efficient and Faithful Task-Agnostic Prompt Compression19 Mar
  8. 2023TCRA-LLM: Token Compression Retrieval Augmented Large Language Model for Inference Cost Reduction24 Oct
  9. 2023LongLLMLingua: Accelerating and Enhancing LLMs in Long Context Scenarios via Prompt Compression10 Oct
  10. 2023Discrete Prompt Compression with Reinforcement Learning17 Aug
  11. 2022Prompt Compression and Contrastive Conditioning for Controllability and Toxicity Reduction in Language Models6 Oct

Other references · 31

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  3. 2024The Llama 3 Herd of Modelscited by 14,715
  4. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  5. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,297
  6. 2016SQuAD: 100,000+ Questions for Machine Comprehension of Textcited by 4,148
  7. 2021Prefix-Tuning: Optimizing Continuous Prompts for Generationcited by 3,915
  8. 2021The Power of Scale for Parameter-Efficient Prompt Tuningcited by 3,795
  9. 2018HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answeringcited by 3,754
  10. 2023QLoRA: Efficient Finetuning of Quantized LLMscited by 3,418
  11. 2020Longformer: The Long-Document Transformercited by 3,413
  12. 2015Neural Machine Translation of Rare Words with Subword Unitscited by 2,918
19 more
  1. 2020The Pile: An 800GB Dataset of Diverse Text for Language Modelingcited by 2,411
  2. 2020Don't Stop Pretraining: Adapt Language Models to Domains and Taskscited by 1,565
  3. 2019Generating Long Sequences with Sparse Transformerscited by 1,546
  4. 2017RACE: Large-scale ReAding Comprehension Dataset From Examinationscited by 861
  5. 2023Vision-Language Models for Vision Tasks: A Surveycited by 839
  6. 2024DoRA: Weight-Decomposed Low-Rank Adaptationcited by 758
  7. 2021Prompt Programming for Large Language Models: Beyond the Few-Shot Paradigmcited by 606
  8. 2023Learning to Compress Prompts with Gist Tokenscited by 390
  9. 2023Adapting Language Models to Compress Contextscited by 356
  10. 2022Reasoning with Language Model Prompting: A Surveycited by 252
  11. 2024Efficient Multimodal Large Language Models: A Surveycited by 96
  12. 2021Combiner: Full Attention Transformer with Sparse Computation Costcited by 64
  13. 2024CompAct: Compressing Retrieved Documents Actively for Question Answeringcited by 64
  14. 2024Learning to Compress Prompt in Natural Language Formatscited by 55
  15. 2024Efficient Prompting Methods for Large Language Models: A Surveycited by 47
  16. 2024A Survey of Prompt Engineering Methods in Large Language Models for Different NLP Taskscited by 47
  17. 2020Inductive Bias and Language Expressivity in Emergent Communicationcited by 6
  18. 2022Extensible Prompts for Language Models on Zero-shot Language Style Customizationcited by 6
  19. 2024UniICL: An Efficient Unified Framework Unifying Compression, Selection, and Generationcited by 6

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 3; 2025: 27; 2026: 37.

67 papers.

Sort:

By area:

  1. 2025LightMem: Lightweight and Efficient Memory-Augmented Generationcited by 180
  2. 2024A Survey on Large Language Model Acceleration based on KV Cache Managementcited by 131
  3. 2025Scaling LLM Multi-turn RL with End-to-end Summarization-based Context Managementcited by 51
  4. 2024MAIN-RAG: Multi-Agent Filtering Retrieval-Augmented Generationcited by 44
  5. 2025LongCodeZip: Compress Long Context for Code Language Modelscited by 39
  6. 2025Rethinking Memory in LLM based Agents: Representations, Operations, and Emerging Topicscited by 36
  7. 2026SkillReducer: Optimizing LLM Agent Skills for Token Efficiencycited by 33
  8. 2025A Survey of Token Compression for Efficient Multimodal Large Language Modelscited by 25
  9. 2025Hierarchical Document Refinement for Long-context Retrieval-augmented Generationcited by 16
  10. 2025Autoencoding-Free Context Compression for LLMs via Contextual Semantic Anchorscited by 14
  11. 2024LightTransfer: Your Long-Context LLM is Secretly a Hybrid Model with Effortless Adaptationcited by 12
  12. 2025R3Mem: Bridging Memory Retention and Retrieval via Reversible Compressioncited by 12
38 more
  1. 2025Enhancing RAG Efficiency with Adaptive Context Compressioncited by 8
  2. 2025Beyond Hard and Soft: Hybrid Context Compression for Balancing Local and Global Information Retentioncited by 7
  3. 2025A Survey on Prompt Tuningcited by 7
  4. 2025ReasonGraph: Visualisation of Reasoning Pathscited by 6
  5. 2025PT-MoE: An Efficient Finetuning Framework for Integrating Mixture-of-Experts into Prompt Tuningcited by 6
  6. 2025Memory Tokens: Large Language Models Can Generate Reversible Sentence Embeddingscited by 6
  7. 2025Semantic Caching of Contextual Summaries for Efficient Question-Answering with Language Modelscited by 5
  8. 2025Agentic Systems in Radiology: Design, Applications, Evaluation, and Challengescited by 5
  9. 2026Compositional Steering of Large Language Models with Steering Tokenscited by 5
  10. 2025MOOSComp: Improving Lightweight Long-Context Compressor via Mitigating Over-Smoothing and Incorporating Outlier Scorescited by 4
  11. 2025Lossless Token Sequence Compression via Meta-Tokenscited by 4
  12. 2025Scaling Up Efficient Small Language Models Serving and Deployment for Semantic Job Searchcited by 3
  13. 2026ShortCoder: Knowledge-Augmented Syntax Optimization for Token-Efficient Code Generationcited by 3
  14. 2026A Guide to Large Language Models in Modeling and Simulation: From Core Techniques to Critical Challengescited by 3
  15. 2026Beyond Position Bias: Shifting Context Compression from Position-Driven to Semantic-Drivencited by 3
  16. 2026Compress the Context, Keep the Commitments: A Formal Framework for Verifiable LLM Context Compressioncited by 3
  17. 2026Mem-π: Adaptive Memory through Learning When and What to Generatecited by 3
  18. 2026ECHO: Prune To Act, Trace To Learn With Selective Turn Memory In Agentic RLcited by 3
  19. 2025Distilling Many-Shot In-Context Learning into a Cheat Sheetcited by 2
  20. 2025Multimodal Policy Internalization for Conversational Agentscited by 2
  21. 2025Analyzing and Internalizing Complex Policy Documents for LLM Agentscited by 2
  22. 2025Script: Graph-Structured and Query-Conditioned Semantic Token Pruning for Multimodal Large Language Modelscited by 2
  23. 2025PAACE: A Plan-Aware Automated Agent Context Engineering Frameworkcited by 2
  24. 2026Fix the Structural Bottleneck: Context Compression via Explicit Information Transmissioncited by 2
  25. 2026Breaking Training Bottlenecks: Effective and Stable Reinforcement Learning for Coding Modelscited by 2
  26. 2026Compression Method Matters: Benchmark-Dependent Output Dynamics in LLM Prompt Compressioncited by 2
  27. 2026Sema Code: Decoupling AI Coding Agents into Programmable, Embeddable Infrastructurecited by 2
  28. 2026AI Agents Do Not Fail Alone:The Context Fails Firstcited by 2
  29. 2025Cmprsr: Abstractive Token-Level Question-Agnostic Prompt Compressorcited by 1
  30. 2025ConCISE: A Reference-Free Conciseness Evaluation Metric for LLM-Generated Answerscited by 1
  31. 2026Data Distribution Matters: A Data-Centric Perspective on Context Compression for Large Language Modelcited by 1
  32. 2026Cognitive Chunking for Soft Prompts: Accelerating Compressor Learning via Block-wise Causal Maskingcited by 1
  33. 2026Breaking the Capability Ceiling of LLM Post-Training by Reintroducing Markov Statescited by 1
  34. 2026DiffuMask: Diffusion Language Model for Token-level Prompt Pruningcited by 1
  35. 2026More Context, Larger Models, or Moral Knowledge? A Systematic Study of Schwartz Value Detection in Political Textscited by 1
  36. 2026CoreMem: Riemannian Retrieval and Fisher-Guided Distillation for Long-Term Memory in Dialogue Agentscited by 1
  37. 2026Fixed RAG Compression Collapses Measured Reader Scalingcited by 1
  38. 2026Lost in Compression: A Controlled Cross-Lingual Audit of Extractive Prompt Compressorscited by 1

50 most cited of 67 All 67 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025Understanding and Improving Information Preservation in Prompt Compression for LLMs
  2. 2025An Empirical Study on Prompt Compression for Large Language Models
  3. 2024Fundamental Limits of Prompt Compression: A Rate-Distortion Framework for Black-Box Language Models
  4. 2023A Survey on Model Compression for Large Language Models
  5. 2026Lexical Prompt Compression for Large Language Models: A Training-Free, Deterministic Pipeline with Empirical Pareto Analysis Across Eleven Task Categories
  6. 2024LanguaShrink: Reducing Token Overhead with Psycholinguistics
  7. 2025Dynamic Compressing Prompts for Efficient Inference of Large Language Models
  8. 2025Better Prompt Compression Without Multi-Layer Perceptrons