# Learning to Compress Prompt in Natural Language Formats

arXiv: https://arxiv.org/abs/2402.18700 · HTML: https://mlsanity.com/paper/2402.18700/

## Key facts

- Cited by 55 papers in the map (data through 14 Oct 2026).
- Builds on 13 of its 23 references that are in the map.
- First posted on arXiv 28 Feb 2024.

## Builds on (most cited first, 13 of 13)

- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [LLaMA: Open and Efficient Foundation Language Models](https://mlsanity.com/paper/2302.13971/) (2023)
- [Training Verifiers to Solve Math Word Problems](https://mlsanity.com/paper/2110.14168/) (2021)
- [PaLM: Scaling Language Modeling with Pathways](https://mlsanity.com/paper/2204.02311/) (2022)
- [Large Language Models are Zero-Shot Reasoners](https://mlsanity.com/paper/2205.11916/) (2022)
- [BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questions](https://mlsanity.com/paper/1905.10044/) (2019)
- [CommonsenseQA: A Question Answering Challenge Targeting Commonsense Knowledge](https://mlsanity.com/paper/1811.00937/) (2018)
- [ERASER: A Benchmark to Evaluate Rationalized NLP Models](https://mlsanity.com/paper/1911.03429/) (2019)
- [Learning to Compress Prompts with Gist Tokens](https://mlsanity.com/paper/2304.08467/) (2023)
- [Solving General Arithmetic Word Problems](https://mlsanity.com/paper/1608.01413/) (2016)
- [Adapting Language Models to Compress Contexts](https://mlsanity.com/paper/2305.14788/) (2023)
- [Prompt Compression and Contrastive Conditioning for Controllability and Toxicity Reduction in Language Models](https://mlsanity.com/paper/2210.03162/) (2022)
- [Context Compression for Auto-regressive Transformers with Sentinel Tokens](https://mlsanity.com/paper/2310.08152/) (2023)

## Built on by (most cited first, 30 of 55)

- [TokenSkip: Controllable Chain-of-Thought Compression in LLMs](https://mlsanity.com/paper/2502.12067/) (2025)
- [LightMem: Lightweight and Efficient Memory-Augmented Generation](https://mlsanity.com/paper/2510.18866/) (2025)
- [Cartridges: Lightweight and general-purpose long context representations via self-study](https://mlsanity.com/paper/2506.06266/) (2025)
- [Prompt Compression for Large Language Models: A Survey](https://mlsanity.com/paper/2410.12388/) (2024)
- [Efficient Prompting Methods for Large Language Models: A Survey](https://mlsanity.com/paper/2404.01077/) (2024)
- [MAIN-RAG: Multi-Agent Filtering Retrieval-Augmented Generation](https://mlsanity.com/paper/2501.00332/) (2024)
- [KV Cache Compression, But What Must We Give in Return? A Comprehensive Benchmark of Long Context Capable Approaches](https://mlsanity.com/paper/2407.01527/) (2024)
- [A Survey of Multimodal Retrieval-Augmented Generation](https://mlsanity.com/paper/2504.08748/) (2025)
- [Perception Compressor: A Training-Free Prompt Compression Framework in Long Context Scenarios](https://mlsanity.com/paper/2409.19272/) (2024)
- [TACO-RL: Task Aware Prompt Compression Optimization with Reinforcement Learning](https://mlsanity.com/paper/2409.13035/) (2024)
- [Chain-of-Thought Compression Should Not Be Blind: V-Skip for Efficient Multimodal Reasoning via Dual-Path Anchoring](https://mlsanity.com/paper/2601.13879/) (2026)
- [Compressing Long Context for Enhancing RAG with AMR-based Concept Distillation](https://mlsanity.com/paper/2405.03085/) (2024)
- [Understanding and Improving Information Preservation in Prompt Compression for LLMs](https://mlsanity.com/paper/2503.19114/) (2025)
- [ProxSparse: Regularized Learning of Semi-Structured Sparsity Masks for Pretrained LLMs](https://mlsanity.com/paper/2502.00258/) (2025)
- [Generalizability of Large Language Model-Based Agents: A Comprehensive Survey](https://mlsanity.com/paper/2509.16330/) (2025)
- [Prompt-MII: Meta-Learning Instruction Induction for LLMs](https://mlsanity.com/paper/2510.16932/) (2025)
- [From Reading to Compressing: Exploring the Multi-document Reader for Prompt Compression](https://mlsanity.com/paper/2410.04139/) (2024)
- [Lossless KV Cache Compression to 2%](https://mlsanity.com/paper/2410.15252/) (2024)
- [Beyond Hard and Soft: Hybrid Context Compression for Balancing Local and Global Information Retention](https://mlsanity.com/paper/2505.15774/) (2025)
- [What Makes a Good Natural Language Prompt?](https://mlsanity.com/paper/2506.06950/) (2025)
- [No Mean Feat: Simple, Strong Baselines for Context Compression](https://mlsanity.com/paper/2510.20797/) (2025)
- [Enhancing Embedding Performance through Large Language Model-based Text Enrichment and Rewriting](https://mlsanity.com/paper/2404.12283/) (2024)
- [Parse Trees Guided LLM Prompt Compression](https://mlsanity.com/paper/2409.15395/) (2024)
- [ProCut: LLM Prompt Compression via Attribution Estimation](https://mlsanity.com/paper/2508.02053/) (2025)
- [CoMeT: Collaborative Memory Transformer for Efficient Long Context Modeling](https://mlsanity.com/paper/2602.01766/) (2026)
- [Do LLMs Benefit From Their Own Words?](https://mlsanity.com/paper/2602.24287/) (2026)
- [Lossless Token Sequence Compression via Meta-Tokens](https://mlsanity.com/paper/2506.00307/) (2025)
- [End-to-End Context Compression at Scale](https://mlsanity.com/paper/2606.09659/) (2026)
- [Scaling Up Efficient Small Language Models Serving and Deployment for Semantic Job Search](https://mlsanity.com/paper/2510.22101/) (2025)
- [ShortCoder: Knowledge-Augmented Syntax Optimization for Token-Efficient Code Generation](https://mlsanity.com/paper/2601.09703/) (2026)

## Abstract (arXiv)

Large language models (LLMs) are great at processing multiple natural language processing tasks, but their abilities are constrained by inferior performance with long context, slow inference speed, and the high cost of computing the results. Deploying LLMs with precise and informative context helps users process large-scale datasets more effectively and cost-efficiently. Existing works rely on compressing long prompt contexts into soft prompts. However, soft prompt compression encounters limitations in transferability across different LLMs, especially API-based LLMs. To this end, this work aims to compress lengthy prompts in the form of natural language with LLM transferability. This poses two challenges: (i) Natural Language (NL) prompts are incompatible with back-propagation, and (ii) NL prompts lack flexibility in imposing length constraints. In this work, we propose a Natural Language Prompt Encapsulation (Nano-Capsulator) framework compressing original prompts into NL formatted Capsule Prompt while maintaining the prompt utility and transferability. Specifically, to tackle the first challenge, the Nano-Capsulator is optimized by a reward function that interacts with the proposed semantics preserving loss. To address the second question, the Nano-Capsulator is optimized by a reward function featuring length constraints. Experimental results demonstrate that the Capsule Prompt can reduce 81.4% of the original length, decrease inference latency up to 4.5x, and save 80.1% of budget overheads while providing transferability across diverse LLMs and different datasets.