mlsanityData to 7 Oct 2026

LLMLingua-2: Data Distillation for Efficient and Faithful Task-Agnostic Prompt Compression

arXiv 2403.12968 ↗First posted 19 Mar 2024cs.CL cs.LGcited by 343

On the map: Efficient LLM inference and serving → Prompt and context compression

Abstract (arXiv)

This paper focuses on task-agnostic prompt compression for better generalizability and efficiency. Considering the redundancy in natural language, existing approaches compress prompts by removing tokens or lexical units according to their information entropy obtained from a causal language model such as LLaMa-7B. The challenge is that information entropy may be a suboptimal compression metric: (i) it only leverages unidirectional context and may fail to capture all essential information needed for prompt compression; (ii) it is not aligned with the prompt compression objective. To address these issues, we propose a data distillation procedure to derive knowledge from an LLM to compress prompts without losing crucial information, and meantime, introduce an extractive text compression dataset. We formulate prompt compression as a token classification problem to guarantee the faithfulness of the compressed prompt to the original one, and use a Transformer encoder as the base architecture to capture all essential information for prompt compression from the full bidirectional context. Our approach leads to lower latency by explicitly learning the compression objective with smaller models such as XLM-RoBERTa-large and mBERT. We evaluate our method on both in-domain and out-of-domain datasets, including MeetingBank, LongBench, ZeroScrolls, GSM8K, and BBH. Despite its small size, our model shows significant performance gains over strong baselines and demonstrates robust generalization ability across different LLMs. Additionally, our model is 3x-6x faster than existing prompt compression methods, while accelerating the end-to-end latency by 1.6x-2.9x with compression ratios of 2x-5x. Our code is available at https://aka.ms/LLMLingua-2.

Key facts. Cited by 343 papers in the map (data through 7 Oct 2026). Placed in Prompt and context compression, one of 12 directions in Efficient LLM inference and serving. Builds on 22 of its 40 references that are in the map. First posted on arXiv 19 Mar 2024.

Builds on

22 of 40 references.

Closest ancestors · 3

Same direction, newest first

  1. 2023LongLLMLingua: Accelerating and Enhancing LLMs in Long Context Scenarios via Prompt Compression10 Oct
  2. 2023Discrete Prompt Compression with Reinforcement Learning17 Aug
  3. 2022Prompt Compression and Contrastive Conditioning for Controllability and Toxicity Reduction in Language Models6 Oct

Other references · 19

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  3. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  4. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,160
  5. 2019Unsupervised Cross-lingual Representation Learning at Scalecited by 3,907
  6. 2023Lost in the Middle: How Language Models Use Long Contextscited by 3,383
  7. 2023Efficient Streaming Language Models with Attention Sinkscited by 2,115
  8. 2022Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language modelscited by 1,970
  9. 2023LongBench: A Bilingual, Multitask Benchmark for Long Context Understandingcited by 1,378
  10. 2023MemGPT: Towards LLMs as Operating Systemscited by 1,295
  11. 2023Scissorhands: Exploiting the Persistence of Importance Hypothesis for LLM KV Cache Compression at Test Timecited by 428
  12. 2023Learning to Compress Prompts with Gist Tokenscited by 379
7 more
  1. 2023Adapting Language Models to Compress Contextscited by 350
  2. 2018WikiHow: A Large Scale Text Summarization Datasetcited by 203
  3. 2023ZeroSCROLLS: A Zero-Shot Benchmark for Long Text Understandingcited by 175
  4. 2023Walking Down the Memory Maze: Beyond Context Limit through Interactive Readingcited by 129
  5. 2018Abstractive Summarization of Reddit Posts with Multi-level Memory Networkscited by 102
  6. 2023MeetingBank: A Benchmark Dataset for Meeting Summarizationcited by 81
  7. 2020DebateSum: A large-scale argument mining and summarization datasetcited by 34

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 50; 2025: 110; 2026: 183.

343 papers.

Sort:

By area:

  1. 2024MInference 1.0: Accelerating Pre-filling for Long-Context LLMs via Dynamic Sparse Attentioncited by 375
  2. 2025TokenSkip: Controllable Chain-of-Thought Compression in LLMscited by 246
  3. 2025LightMem: Lightweight and Efficient Memory-Augmented Generationcited by 163
  4. 2024xRAG: Extreme Context Compression for Retrieval-augmented Generation with One Tokencited by 123
  5. 2025FlexPrefill: A Context-Aware Sparse Attention Mechanism for Efficient Long-Sequence Inferencecited by 121
  6. 2024C3oT: Generating Shorter Chain-of-Thought without Compromising Effectivenesscited by 117
  7. 2025On Memory Construction and Retrieval for Personalized Conversational Agentscited by 101
  8. 2024PQCache: Product Quantization-based KVCache for Long Context LLM Inferencecited by 93
  9. 2025Cartridges: Lightweight and general-purpose long context representations via self-studycited by 74
  10. 2024Prompt Compression for Large Language Models: A Surveycited by 65
  11. 2024CompAct: Compressing Retrieved Documents Actively for Question Answeringcited by 64
  12. 2025KVLink: Accelerating Large Language Models via Efficient KV Cache Reusecited by 60
38 more
  1. 2024SCBench: A KV Cache-Centric Analysis of Long-Context Methodscited by 59
  2. 2024Prompt Compression with Context-Aware Sentence Encoding for Fast and Improved LLM Inferencecited by 51
  3. 2024Efficient Prompting Methods for Large Language Models: A Surveycited by 46
  4. 2024500xCompressor: Generalized Prompt Compression for Large Language Modelscited by 45
  5. 2025Reducing Cost of LLM Agents with Trajectory Reductioncited by 44
  6. 2024KV Cache Compression, But What Must We Give in Return? A Comprehensive Benchmark of Long Context Capable Approachescited by 43
  7. 2024Merge, Ensemble, and Cooperate! A Survey on Collaborative Strategies in the Era of Large Language Modelscited by 43
  8. 2026SWE-Pruner: Self-Adaptive Context Pruning for Coding Agentscited by 40
  9. 2024Discovering the Gems in Early Layers: Accelerating Long-Context LLMs with 1000x Input Token Reductioncited by 39
  10. 2025LongCodeZip: Compress Long Context for Code Language Modelscited by 38
  11. 2025Context as a Tool: Context Management for Long-Horizon SWE-Agentscited by 38
  12. 2025A Survey of Multimodal Retrieval-Augmented Generationcited by 37
  13. 2026Doc-to-LoRA: Learning to Instantly Internalize Contextscited by 36
  14. 2024How Much Can RAG Help the Reasoning of LLM?cited by 31
  15. 2024Customized Retrieval Augmented Generation and Benchmarking for EDA Tool Documentation QAcited by 28
  16. 2025Cramming 1568 Tokens into a Single Vector and Back Again: Exploring the Limits of Embedding Space Capacitycited by 27
  17. 2024EfficientRAG: Efficient Retriever for Multi-Hop Question Answeringcited by 24
  18. 2024TACO-RL: Task Aware Prompt Compression Optimization with Reinforcement Learningcited by 22
  19. 2024Perception Compressor: A Training-Free Prompt Compression Framework in Long Context Scenarioscited by 22
  20. 2025A Survey of Token Compression for Efficient Multimodal Large Language Modelscited by 22
  21. 2025Vision-centric Token Compression in Large Language Modelcited by 21
  22. 2025DAST: Context-Aware Compression in LLMs via Dynamic Allocation of Soft Tokenscited by 21
  23. 2026Chain-of-Thought Compression Should Not Be Blind: V-Skip for Efficient Multimodal Reasoning via Dual-Path Anchoringcited by 21
  24. 2025PISCO: Pretty Simple Compression for Retrieval-Augmented Generationcited by 20
  25. 2025LEMMA: Learning from Errors for MatheMatical Advancement in LLMscited by 20
  26. 2025GMSA: Enhancing Context Compression via Group Merging and Layer Semantic Alignmentcited by 20
  27. 2025Not All Tokens Are What You Need In Thinkingcited by 20
  28. 2025SwiReasoning: Switch-Thinking in Latent and Explicit for Pareto-Superior Reasoning LLMscited by 20
  29. 2026Beyond Static Summarization: Proactive Memory Extraction for LLM Agentscited by 20
  30. 2025AttentionRAG: Attention-Guided Context Pruning in Retrieval-Augmented Generationcited by 19
  31. 2025Can Pruning Improve Reasoning? Revisiting Long-CoT Compression with Capability in Mind for Better Reasoningcited by 19
  32. 2024Fundamental Limits of Prompt Compression: A Rate-Distortion Framework for Black-Box Language Modelscited by 18
  33. 2024EXIT: Context-Aware Extractive Compression for Enhancing Retrieval-Augmented Generationcited by 17
  34. 2025Sculptor: Empowering LLMs with Cognitive Agency via Active Context Managementcited by 17
  35. 2024LLoCO: Learning Long Contexts Offlinecited by 16
  36. 2024Refiner: Restructure Retrieval Content Efficiently to Advance Question-Answering Capabilitiescited by 16
  37. 2025InfiniteICL: Breaking the Limit of Context Window Size via Long Short-term Memory Transformationcited by 16
  38. 2025Hierarchical Document Refinement for Long-context Retrieval-augmented Generationcited by 16

50 most cited of 343 All 343 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2026Lexical Prompt Compression for Large Language Models: A Training-Free, Deterministic Pipeline with Empirical Pareto Analysis Across Eleven Task Categories
  2. 2024Prompt-SAW: Leveraging Relation-Aware Graphs for Textual Prompt Compression
  3. 2026Lossless Prompt Compression via Dictionary-Encoding and In-Context Learning: Enabling Cost-Effective LLM Analysis of Repetitive Data
  4. 2023LLMLingua: Compressing Prompts for Accelerated Inference of Large Language Models
  5. 2024Style-Compress: An LLM-Based Prompt Compression Framework Considering Task-Specific Styles
  6. 2026Prompt Compression via Activation Aggregation
  7. 2025Better Prompt Compression Without Multi-Layer Perceptrons
  8. 2024Characterizing Prompt Compression Methods for Long Context Inference