mlsanityData to 7 Oct 2026

500xCompressor: Generalized Prompt Compression for Large Language Models

arXiv 2408.03094 ↗First posted 6 Aug 2024cs.CLcited by 45

On the map: Efficient LLM inference and serving → Prompt and context compression

Abstract (arXiv)

Prompt compression is crucial for enhancing inference speed, reducing costs, and improving user experience. However, current methods face challenges such as low compression ratios and potential data leakage during evaluation. To address these issues, we propose 500xCompressor, a method that compresses extensive natural language contexts into a minimum of one single special token. The 500xCompressor introduces approximately 0.3% additional parameters and achieves compression ratios ranging from 6x to 480x. It is designed to compress any text, answer various types of questions, and could be utilized by the original large language model (LLM) without requiring fine-tuning. Initially, 500xCompressor was pretrained on the Arxiv Corpus, followed by fine-tuning on the ArxivQA dataset, and subsequently evaluated on strictly unseen and classical question answering (QA) datasets. The results demonstrate that the LLM retained 62.26-72.89% of its capabilities compared to using non-compressed prompts. This study also shows that not all the compressed tokens are equally utilized and that K V values have significant advantages over embeddings in preserving information at high compression ratios. The highly compressive nature of natural language prompts, even for fine-grained complex information, suggests promising potential for future applications and further research into developing a new LLM language.

Key facts. Cited by 45 papers in the map (data through 7 Oct 2026). Placed in Prompt and context compression, one of 12 directions in Efficient LLM inference and serving. Builds on 17 of its 26 references that are in the map. First posted on arXiv 6 Aug 2024.

Builds on

17 of 26 references.

Closest ancestors · 6

Same direction, newest first

  1. 2024Context Embeddings for Efficient Answer Generation in RAG12 Jul
  2. 2024xRAG: Extreme Context Compression for Retrieval-augmented Generation with One Token22 May
  3. 2024LLMLingua-2: Data Distillation for Efficient and Faithful Task-Agnostic Prompt Compression19 Mar
  4. 2023Context Compression for Auto-regressive Transformers with Sentinel Tokens12 Oct
  5. 2023LongLLMLingua: Accelerating and Enhancing LLMs in Long Context Scenarios via Prompt Compression10 Oct
  6. 2022Prompt Compression and Contrastive Conditioning for Controllability and Toxicity Reduction in Language Models6 Oct

Other references · 11

  1. 2024The Llama 3 Herd of Modelscited by 14,192
  2. 2016SQuAD: 100,000+ Questions for Machine Comprehension of Textcited by 4,115
  3. 2021Prefix-Tuning: Optimizing Continuous Prompts for Generationcited by 3,878
  4. 2021The Power of Scale for Parameter-Efficient Prompt Tuningcited by 3,769
  5. 2017TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehensioncited by 2,895
  6. 2017RACE: Large-scale ReAding Comprehension Dataset From Examinationscited by 851
  7. 2017Zero-Shot Relation Extraction via Reading Comprehensioncited by 425
  8. 2023Learning to Compress Prompts with Gist Tokenscited by 379
  9. 2023Adapting Language Models to Compress Contextscited by 350
  10. 2019MRQA 2019 Shared Task: Evaluating Generalization in Reading Comprehensioncited by 183
  11. 2024Tag-LLM: Repurposing General-Purpose LLMs for Specialized Domainscited by 30

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 1; 2025: 21; 2026: 23.

45 papers.

Sort:

By area:

  1. 2025LongCodeZip: Compress Long Context for Code Language Modelscited by 38
  2. 2025A Survey of Multimodal Retrieval-Augmented Generationcited by 37
  3. 2025PISCO: Pretty Simple Compression for Retrieval-Augmented Generationcited by 20
  4. 2025GMSA: Enhancing Context Compression via Group Merging and Layer Semantic Alignmentcited by 20
  5. 2025Autoencoding-Free Context Compression for LLMs via Contextual Semantic Anchorscited by 14
  6. 2026COMI: Coarse-to-fine Context Compression via Marginal Information Gaincited by 13
  7. 2026Latent Context Compilation: Distilling Long Context into Compact Portable Memorycited by 11
  8. 2025Embedding-to-Prefix: Parameter-Efficient Personalization for Pre-Trained Large Language Modelscited by 10
  9. 2025Continuous Autoregressive Language Modelscited by 9
  10. 2025Prompt-MII: Meta-Learning Instruction Induction for LLMscited by 8
  11. 2025Beyond Hard and Soft: Hybrid Context Compression for Balancing Local and Global Information Retentioncited by 7
  12. 2025A Survey on Prompt Tuningcited by 7
33 more
  1. 2024PromptExp: Multi-granularity Prompt Explanation of Large Language Modelscited by 6
  2. 2025ReasonGraph: Visualisation of Reasoning Pathscited by 6
  3. 2025PT-MoE: An Efficient Finetuning Framework for Integrating Mixture-of-Experts into Prompt Tuningcited by 6
  4. 2025No Mean Feat: Simple, Strong Baselines for Context Compressioncited by 6
  5. 2026GradMem: Learning to Write Context into Memory with Test-Time Gradient Descentcited by 6
  6. 2025LoRA-Gen: Specializing Large Language Model via Online LoRA Generationcited by 5
  7. 2025CompLLM: Compression for Long Context Q&Acited by 5
  8. 2026Toward Reliable Context Compression for Long-Horizon Agents: An Empirical Study of Execution Instabilitycited by 5
  9. 2025Better Prompt Compression Without Multi-Layer Perceptronscited by 4
  10. 2025Knowing When to Stop: Efficient Context Processing via Latent Sufficiency Signalscited by 4
  11. 2025Lossless Token Sequence Compression via Meta-Tokenscited by 4
  12. 2026Large Language Model as Token Compressor and Decompressorcited by 4
  13. 2026LKV: End-to-End Learning of Head-wise Budgets and Token Selection for LLM KV Cache Evictioncited by 4
  14. 2026End-to-End Context Compression at Scalecited by 4
  15. 2025BatchGEMBA: Token-Efficient Machine Translation Evaluation with Batched Prompting and Prompt Compressioncited by 3
  16. 2025From Context to EDUs: Faithful and Structured Context Compression via Elementary Discourse Unit Decompositioncited by 3
  17. 2026ShortCoder: Knowledge-Augmented Syntax Optimization for Token-Efficient Code Generationcited by 3
  18. 2026Beyond Position Bias: Shifting Context Compression from Position-Driven to Semantic-Drivencited by 3
  19. 2026Fix the Structural Bottleneck: Context Compression via Explicit Information Transmissioncited by 2
  20. 2026Breaking Training Bottlenecks: Effective and Stable Reinforcement Learning for Coding Modelscited by 2
  21. 2026AutoScientist-Quant: Self-Evolving Coding Agents for Automatic Research in Quantitative Investmentcited by 2
  22. 2025Optical Context Compression Is Just (Bad) Autoencodingcited by 1
  23. 2026Data Distribution Matters: A Data-Centric Perspective on Context Compression for Large Language Modelcited by 1
  24. 2026Cache-Aware Prompt Compression:A Two-Tier Cost Model for LLM API Cachingcited by 1
  25. 2026Cartridges++: KV Cache Compression without Off-Context Derailmentcited by 1
  26. 2026Density-aware Soft Context Compression with Semi-Dynamic Compression Ratiocited by 0
  27. 2026A Unified Model and Document Representation for On-Device Retrieval-Augmented Generationcited by 0
  28. 2026HybridThinker: Efficient Chain-of-Thought Reasoning via Compressed Memory and Transient Thought Stepscited by 0
  29. 2026Compression Beyond the Uncompressed: A Two-Stage Training Recipe for Soft Context Compression in RAGcited by 0
  30. 2026Dynamic Semantic Compression for Efficient Latent-Space Inference in Large Language Modelscited by 0
  31. 2026MORSE: Multi-Context Ordering via Reverse Scoring for Evidence-Preserving Compressioncited by 0
  32. 2026Compress What You See, Not What You Say: Anchored Context Distillation for Latent-Observation Software Engineering Agentscited by 0
  33. 2026Doc2LoRA Provides Decodable Representations of Scientific Ideascited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Prompt Compression with Context-Aware Sentence Encoding for Fast and Improved LLM Inference
  2. 2025Cmprsr: Abstractive Token-Level Question-Agnostic Prompt Compressor
  3. 2024Prompt Compression for Large Language Models: A Survey
  4. 2024Fundamental Limits of Prompt Compression: A Rate-Distortion Framework for Black-Box Language Models
  5. 2023LLMLingua: Compressing Prompts for Accelerated Inference of Large Language Models
  6. 2025An Empirical Study on Prompt Compression for Large Language Models
  7. 2026Prompt Compression in the Wild: Measuring Latency, Rate Adherence, and Quality for Faster LLM Inference
  8. 2026Compressing Sequences in the Latent Embedding Space: K-Token Merging for Large Language Models