mlsanityData to 7 Oct 2026

Learning to Compress Prompt in Natural Language Formats

arXiv 2402.18700 ↗First posted 28 Feb 2024cs.CL cs.AI cs.LGcited by 54

Not on the map. Why?

Cited by 25 in Efficient LLM inference and serving, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Large language models (LLMs) are great at processing multiple natural language processing tasks, but their abilities are constrained by inferior performance with long context, slow inference speed, and the high cost of computing the results. Deploying LLMs with precise and informative context helps users process large-scale datasets more effectively and cost-efficiently. Existing works rely on compressing long prompt contexts into soft prompts. However, soft prompt compression encounters limitations in transferability across different LLMs, especially API-based LLMs. To this end, this work aims to compress lengthy prompts in the form of natural language with LLM transferability. This poses two challenges: (i) Natural Language (NL) prompts are incompatible with back-propagation, and (ii) NL prompts lack flexibility in imposing length constraints. In this work, we propose a Natural Language Prompt Encapsulation (Nano-Capsulator) framework compressing original prompts into NL formatted Capsule Prompt while maintaining the prompt utility and transferability. Specifically, to tackle the first challenge, the Nano-Capsulator is optimized by a reward function that interacts with the proposed semantics preserving loss. To address the second question, the Nano-Capsulator is optimized by a reward function featuring length constraints. Experimental results demonstrate that the Capsule Prompt can reduce 81.4% of the original length, decrease inference latency up to 4.5x, and save 80.1% of budget overheads while providing transferability across diverse LLMs and different datasets.

Key facts. Cited by 54 papers in the map (data through 7 Oct 2026). Builds on 13 of its 23 references that are in the map. First posted on arXiv 28 Feb 2024.

Builds on

13 of 23 references.

References · 13

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  3. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  4. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
  5. 2022Large Language Models are Zero-Shot Reasonerscited by 4,323
  6. 2019BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questionscited by 2,140
  7. 2018CommonsenseQA: A Question Answering Challenge Targeting Commonsense Knowledgecited by 1,793
  8. 2019ERASER: A Benchmark to Evaluate Rationalized NLP Modelscited by 523
  9. 2023Learning to Compress Prompts with Gist Tokenscited by 379
  10. 2016Solving General Arithmetic Word Problemscited by 353
  11. 2023Adapting Language Models to Compress Contextscited by 350
  12. 2022Prompt Compression and Contrastive Conditioning for Controllability and Toxicity Reduction in Language Modelscited by 96
1 more
  1. 2023Context Compression for Auto-regressive Transformers with Sentinel Tokenscited by 8

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 11; 2025: 23; 2026: 20.

54 papers.

Sort:

By area:

  1. 2025TokenSkip: Controllable Chain-of-Thought Compression in LLMscited by 246
  2. 2025LightMem: Lightweight and Efficient Memory-Augmented Generationcited by 163
  3. 2025Cartridges: Lightweight and general-purpose long context representations via self-studycited by 74
  4. 2024Prompt Compression for Large Language Models: A Surveycited by 65
  5. 2024Efficient Prompting Methods for Large Language Models: A Surveycited by 46
  6. 2024KV Cache Compression, But What Must We Give in Return? A Comprehensive Benchmark of Long Context Capable Approachescited by 43
  7. 2024MAIN-RAG: Multi-Agent Filtering Retrieval-Augmented Generationcited by 43
  8. 2025A Survey of Multimodal Retrieval-Augmented Generationcited by 37
  9. 2024TACO-RL: Task Aware Prompt Compression Optimization with Reinforcement Learningcited by 22
  10. 2024Perception Compressor: A Training-Free Prompt Compression Framework in Long Context Scenarioscited by 22
  11. 2026Chain-of-Thought Compression Should Not Be Blind: V-Skip for Efficient Multimodal Reasoning via Dual-Path Anchoringcited by 21
  12. 2024Compressing Long Context for Enhancing RAG with AMR-based Concept Distillationcited by 16
38 more
  1. 2025Understanding and Improving Information Preservation in Prompt Compression for LLMscited by 13
  2. 2025ProxSparse: Regularized Learning of Semi-Structured Sparsity Masks for Pretrained LLMscited by 10
  3. 2025Generalizability of Large Language Model-Based Agents: A Comprehensive Surveycited by 10
  4. 2025Prompt-MII: Meta-Learning Instruction Induction for LLMscited by 8
  5. 2024From Reading to Compressing: Exploring the Multi-document Reader for Prompt Compressioncited by 7
  6. 2024Lossless KV Cache Compression to 2%cited by 7
  7. 2025Beyond Hard and Soft: Hybrid Context Compression for Balancing Local and Global Information Retentioncited by 7
  8. 2024Enhancing Embedding Performance through Large Language Model-based Text Enrichment and Rewritingcited by 6
  9. 2024Parse Trees Guided LLM Prompt Compressioncited by 6
  10. 2025What Makes a Good Natural Language Prompt?cited by 6
  11. 2025No Mean Feat: Simple, Strong Baselines for Context Compressioncited by 6
  12. 2025ProCut: LLM Prompt Compression via Attribution Estimationcited by 5
  13. 2026CoMeT: Collaborative Memory Transformer for Efficient Long Context Modelingcited by 5
  14. 2025Lossless Token Sequence Compression via Meta-Tokenscited by 4
  15. 2026Do LLMs Benefit From Their Own Words?cited by 4
  16. 2026End-to-End Context Compression at Scalecited by 4
  17. 2025Scaling Up Efficient Small Language Models Serving and Deployment for Semantic Job Searchcited by 3
  18. 2026ShortCoder: Knowledge-Augmented Syntax Optimization for Token-Efficient Code Generationcited by 3
  19. 2026ReasonCACHE: Teaching LLMs To Reason Without Weight Updatescited by 3
  20. 2025Lost in the Passage: Passage-level In-context Learning Does Not Necessarily Need a "Passage"cited by 2
  21. 2025Learned Structure in Cartridges: Keys as Shareable Routers in Self-Studied Representationscited by 2
  22. 2025Multimodal Policy Internalization for Conversational Agentscited by 2
  23. 2025CompactPrompt: A Unified Pipeline for Prompt Data Compression in LLM Workflowscited by 2
  24. 2025Analyzing and Internalizing Complex Policy Documents for LLM Agentscited by 1
  25. 2025FrugalPrompt: Reducing Contextual Overhead in Large Language Models via Token Attributioncited by 1
  26. 2025Optical Context Compression Is Just (Bad) Autoencodingcited by 1
  27. 2026ACR: Adaptive Context Refactoring via Context Refactoring Operators for Multi-Turn Dialoguecited by 1
  28. 2026Token Sparse Attention: Efficient Long-Context Inference with Interleaved Token Selectioncited by 1
  29. 2026Cognitive Chunking for Soft Prompts: Accelerating Compressor Learning via Block-wise Causal Maskingcited by 1
  30. 2026Multilinguality at the Edge: Developing Language Models for the Global Southcited by 1
  31. 2026Large Language Models Do Not Always Need Readable Languagecited by 1
  32. 2026IterCOMP: Reasoning-aware Adaptive Prompt Compression for Multi-hop Question Answeringcited by 1
  33. 2025When Compression Becomes an Attack Surface: Black-Box Attacks on Prompt-Compressed LLM Agentscited by 0
  34. 2025Cmprsr: Abstractive Token-Level Question-Agnostic Prompt Compressorcited by 0
  35. 2026CoCR-RAG: Enhancing Retrieval-Augmented Generation in Web Q&A via Concept-oriented Context Reconstructioncited by 0
  36. 2026A Unified Model and Document Representation for On-Device Retrieval-Augmented Generationcited by 0
  37. 2026Compressing Sequences in the Latent Embedding Space: K-Token Merging for Large Language Modelscited by 0
  38. 2026Every Time I Hire a Linguist, Inference Costs Go Down: On Linguistic Rules as Effective Prompt Compressorscited by 0

50 most cited of 54 All 54 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025An Empirical Study on Prompt Compression for Large Language Models
  2. 2025Dynamic Compressing Prompts for Efficient Inference of Large Language Models
  3. 2024Adapting LLMs for Efficient Context Processing through Soft Prompt Compression
  4. 2023LLMLingua: Compressing Prompts for Accelerated Inference of Large Language Models
  5. 2024Fundamental Limits of Prompt Compression: A Rate-Distortion Framework for Black-Box Language Models
  6. 2026Lexical Prompt Compression for Large Language Models: A Training-Free, Deterministic Pipeline with Empirical Pareto Analysis Across Eleven Task Categories
  7. 2023LongLLMLingua: Accelerating and Enhancing LLMs in Long Context Scenarios via Prompt Compression
  8. 2024LanguaShrink: Reducing Token Overhead with Psycholinguistics