mlsanityData to 14 Oct 2026

An Empirical Study on Prompt Compression for Large Language Models

arXiv 2505.00019 ↗First posted 24 Apr 2025cs.CL cs.AIcited by 8

On the map: Efficient LLM inference and serving → Prompt and context compression

Abstract (arXiv)

Prompt engineering enables Large Language Models (LLMs) to perform a variety of tasks. However, lengthy prompts significantly increase computational complexity and economic costs. To address this issue, we study six prompt compression methods for LLMs, aiming to reduce prompt length while maintaining LLM response quality. In this paper, we present a comprehensive analysis covering aspects such as generation performance, model hallucinations, efficacy in multimodal tasks, word omission analysis, and more. We evaluate these methods across 13 datasets, including news, scientific articles, commonsense QA, math QA, long-context QA, and VQA datasets. Our experiments reveal that prompt compression has a greater impact on LLM performance in long contexts compared to short ones. In the Longbench evaluation, moderate compression even enhances LLM performance. Our code and data is available at https://github.com/3DAgentWorld/Toolkit-for-Prompt-Compression.

Key facts. Cited by 8 papers in the map (data through 14 Oct 2026). Placed in Prompt and context compression, one of 12 directions in Efficient LLM inference and serving. Builds on 39 of its 50 references that are in the map. First posted on arXiv 24 Apr 2025.

Builds on

39 of 50 references.

Closest ancestors · 3

Same direction, newest first

  1. 2024LLMLingua-2: Data Distillation for Efficient and Faithful Task-Agnostic Prompt Compression19 Mar
  2. 2023LongLLMLingua: Accelerating and Enhancing LLMs in Long Context Scenarios via Prompt Compression10 Oct
  3. 2023In-context Autoencoder for Context Compression in a Large Language Model13 Jul

Other references · 36

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  2. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,459
  3. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  4. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,297
  5. 2019BERTScore: Evaluating Text Generation with BERTcited by 5,512
  6. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,673
  7. 2023Lost in the Middle: How Language Models Use Long Contextscited by 3,455
  8. 2021Efficiently Modeling Long Sequences with Structured State Spacescited by 2,465
  9. 2023Efficient Streaming Language Models with Attention Sinkscited by 2,186
  10. 2022Survey of Hallucination in Natural Language Generationcited by 2,075
  11. 2022Challenging BIG-Bench Tasks and Whether Chain-of-Thought Can Solve Themcited by 1,807
  12. 2020Linformer: Self-Attention with Linear Complexitycited by 1,474
24 more
  1. 2023LongBench: A Bilingual, Multitask Benchmark for Long Context Understandingcited by 1,417
  2. 2019OK-VQA: A Visual Question Answering Benchmark Requiring External Knowledgecited by 1,121
  3. 2018Self-Attention with Relative Position Representationscited by 1,120
  4. 2023Vision Transformers Need Registerscited by 871
  5. 2023H2O: Heavy-Hitter Oracle for Efficient Generative Inference of Large Language Modelscited by 832
  6. 2023LLM-Pruner: On the Structural Pruning of Large Language Modelscited by 650
  7. 2023Scissorhands: Exploiting the Persistence of Importance Hypothesis for LLM KV Cache Compression at Test Timecited by 439
  8. 2023Resurrecting Recurrent Neural Networks for Long Sequencescited by 408
  9. 2015A Neural Attention Model for Abstractive Sentence Summarizationcited by 396
  10. 2023Studying Large Language Model Generalization with Influence Functionscited by 326
  11. 2023A Long Way to Go: Investigating Length Correlations in RLHFcited by 291
  12. 2023LLMLingua: Compressing Prompts for Accelerated Inference of Large Language Modelscited by 287
  13. 2021IconQA: A New Benchmark for Abstract Diagram Understanding and Visual Language Reasoningcited by 267
  14. 2023Compressing Context to Enhance Inference Efficiency of Large Language Modelscited by 189
  15. 2021OpenPrompt: An Open-source Framework for Prompt-learningcited by 148
  16. 2023ChainForge: A Visual Toolkit for Prompt Engineering and LLM Hypothesis Testingcited by 88
  17. 2024The Dawn After the Dark: An Empirical Study on Factuality Hallucination in Large Language Modelscited by 88
  18. 2023Response Length Perception and Sequence Scheduling: An LLM-Empowered LLM Inference Pipelinecited by 86
  19. 2024Beyond the Limits: A Survey of Techniques to Extend the Context Length in Large Language Modelscited by 76
  20. 2019Lightweight and Efficient End-to-End Speech Recognition Using Low-Rank Transformercited by 47
  21. 2021Keep it Simple: Unsupervised Simplification of Multi-Paragraph Textcited by 45
  22. 2021A Simple and Effective Positional Encoding for Transformerscited by 37
  23. 2024Unveiling the Generalization Power of Fine-Tuned Large Language Modelscited by 36
  24. 2022Efficient Unsupervised Sentence Compression by Fine-tuning Transformers with Reinforcement Learningcited by 18

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 2; 2026: 6.

8 papers.

  1. 2026A Guide to Large Language Models in Modeling and Simulation: From Core Techniques to Critical Challengescited by 3
  2. 2025CompactPrompt: A Unified Pipeline for Prompt Data Compression in LLM Workflowscited by 2
  3. 2026Prompt Compression in Diffusion Large Language Models: Evaluating LLMLingua-2 on LLaDAcited by 1
  4. 2025PaceLLM: Brain-Inspired Large Language Models for Long-Context Understandingcited by 0
  5. 2026CROP: Token-Efficient Reasoning in Large Language Models via Regularized Prompt Optimizationcited by 0
  6. 2026Implicit Framing in Obstetric Counseling Notes: A Grounded LLM Pipeline on a VBAC-Eligible Cohortcited by 0
  7. 2026MuMuTestUp: Mutation-based Multi-Agent Test Case Updatecited by 0
  8. 2026LAM: Efficient Lossy Agent Memory Framework With A Retrieval-Score Error Boundcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Prompt Compression for Large Language Models: A Survey
  2. 2025Understanding and Improving Information Preservation in Prompt Compression for LLMs
  3. 2026Prompt Compression in the Wild: Measuring Latency, Rate Adherence, and Quality for Faster LLM Inference
  4. 2023A Survey on Model Compression for Large Language Models
  5. 2024Fundamental Limits of Prompt Compression: A Rate-Distortion Framework for Black-Box Language Models
  6. 2025Dynamic Compressing Prompts for Efficient Inference of Large Language Models
  7. 2024Efficient Prompting Methods for Large Language Models: A Survey
  8. 2023Compress, Then Prompt: Improving Accuracy-Efficiency Trade-off of LLM Inference with Transferable Prompt