mlsanityData to 14 Oct 2026

LLMLingua: Compressing Prompts for Accelerated Inference of Large Language Models

arXiv 2310.05736 ↗First posted 9 Oct 2023cs.CL cs.LGcited by 287

Not on the map. Why?

Cited by 80 in Efficient LLM inference and serving, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Large language models (LLMs) have been applied in various applications due to their astonishing capabilities. With advancements in technologies such as chain-of-thought (CoT) prompting and in-context learning (ICL), the prompts fed to LLMs are becoming increasingly lengthy, even exceeding tens of thousands of tokens. To accelerate model inference and reduce cost, this paper presents LLMLingua, a coarse-to-fine prompt compression method that involves a budget controller to maintain semantic integrity under high compression ratios, a token-level iterative compression algorithm to better model the interdependence between compressed contents, and an instruction tuning based method for distribution alignment between language models. We conduct experiments and analysis over four datasets from different scenarios, i.e., GSM8K, BBH, ShareGPT, and Arxiv-March23; showing that the proposed approach yields state-of-the-art performance and allows for up to 20x compression with little performance loss. Our code is available at https://aka.ms/LLMLingua.

Key facts. Cited by 287 papers in the map (data through 14 Oct 2026). Builds on 27 of its 47 references that are in the map. First posted on arXiv 9 Oct 2023.

Builds on

27 of 47 references.

References · 27

  1. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
  2. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,459
  3. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  4. 2019BERTScore: Evaluating Text Generation with BERTcited by 5,512
  5. 2019XLNet: Generalized Autoregressive Pretraining for Language Understandingcited by 2,811
  6. 2022Challenging BIG-Bench Tasks and Whether Chain-of-Thought Can Solve Themcited by 1,807
  7. 2022SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Modelscited by 1,386
  8. 2023WizardLM: Empowering large pre-trained language models to follow complex instructionscited by 1,125
  9. 2023SparseGPT: Massive Language Models Can Be Accurately Pruned in One-Shotcited by 1,121
  10. 2022Token Merging: Your ViT But Fastercited by 1,021
  11. 2022LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scalecited by 906
  12. 2021DynamicViT: Efficient Vision Transformers with Dynamic Token Sparsificationcited by 864
15 more
  1. 2023Learning to Compress Prompts with Gist Tokenscited by 390
  2. 2022Complexity-Based Prompting for Multi-Step Reasoningcited by 388
  3. 2023Adapting Language Models to Compress Contextscited by 356
  4. 2021Learned Token Pruning for Transformerscited by 139
  5. 2023Chain-of-Thought Hub: A Continuous Effort to Measure Large Language Models' Reasoning Performancecited by 102
  6. 2022Prompt Compression and Contrastive Conditioning for Controllability and Toxicity Reduction in Language Modelscited by 97
  7. 2021Types of Out-of-Distribution Texts and How to Detect Themcited by 80
  8. 2020Length-Adaptive Transformer: Train Once with Length Drop, Use Anytime with Searchcited by 76
  9. 2023MLCopilot: Unleashing the Power of Large Language Models in Solving Machine Learning Taskscited by 66
  10. 2023Inference with Reference: Lossless Acceleration of Large Language Modelscited by 65
  11. 2023Unlocking Context Constraints of LLMs: Enhancing Context Efficiency of LLMs with Self-Information-Based Content Filteringcited by 65
  12. 2022AdapLeR: Speeding up Inference by Adaptive Length Reductioncited by 19
  13. 2023Semantic Compression With Large Language Modelscited by 17
  14. 2022Multi-Level Knowledge Distillation for Out-of-Distribution Detection in Textcited by 11
  15. 2022Self-Supervised Losses for One-Class Textual Anomaly Detectioncited by 5

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 6; 2024: 64; 2025: 75; 2026: 142.

287 papers.

Sort:

By area:

  1. 2023Retrieval-Augmented Generation for Large Language Models: A Surveycited by 2,212
  2. 2023A Survey on Hallucination in Large Language Models: Principles, Taxonomy, Challenges, and Open Questionscited by 1,601
  3. 2024LongMemEval: Benchmarking Chat Assistants on Long-Term Interactive Memorycited by 707
  4. 2023LongLLMLingua: Accelerating and Enhancing LLMs in Long Context Scenarios via Prompt Compressioncited by 453
  5. 2025Native Sparse Attention: Hardware-Aligned and Natively Trainable Sparse Attentioncited by 407
  6. 2024MInference 1.0: Accelerating Pre-filling for Long-Context LLMs via Dynamic Sparse Attentioncited by 383
  7. 2024Retrieval-Augmented Generation for AI-Generated Content: A Surveycited by 354
  8. 2025MemAgent: Reshaping Long-Context LLM with Multi-Conv RL-based Memory Agentcited by 246
  9. 2023PromptAgent: Strategic Planning with Language Models Enables Expert-level Prompt Optimizationcited by 233
  10. 2024CacheBlend: Fast Large Language Model Serving for RAG with Cached Knowledge Fusioncited by 223
  11. 2024A Survey on Efficient Inference for Large Language Modelscited by 205
  12. 2025Large Language Model Agent: A Survey on Methodology, Applications and Challengescited by 205
38 more
  1. 2023CacheGen: KV Cache Compression and Streaming for Fast Large Language Model Servingcited by 203
  2. 2024Compressed Chain of Thought: Efficient Reasoning Through Dense Representationscited by 201
  3. 2024FlashRAG: A Modular Toolkit for Efficient Retrieval-Augmented Generation Researchcited by 166
  4. 2024Mobile Edge Intelligence for Large Language Models: A Contemporary Surveycited by 128
  5. 2025On Memory Construction and Retrieval for Personalized Conversational Agentscited by 105
  6. 2024Long Context Compression with Activation Beaconcited by 104
  7. 2024InstructRAG: Instructing Retrieval-Augmented Generation via Self-Synthesized Rationalescited by 101
  8. 2024Keep the Cost Down: A Review on Methods to Optimize LLM' s KV-Cache Consumptioncited by 97
  9. 2024Rethinking LLM Memorization through the Lens of Adversarial Compressioncited by 94
  10. 2024From Matching to Generation: A Survey on Generative Information Retrievalcited by 89
  11. 2025Cartridges: Lightweight and general-purpose long context representations via self-studycited by 86
  12. 2024Searching for Best Practices in Retrieval-Augmented Generationcited by 83
  13. 2025Lifelong Learning of Large Language Model based Agents: A Roadmapcited by 83
  14. 2024From Isolated Conversations to Hierarchical Schemas: Dynamic Tree Memory Representation for LLMscited by 79
  15. 2024Beyond the Limits: A Survey of Techniques to Extend the Context Length in Large Language Modelscited by 76
  16. 2023Unleashing the potential of prompt engineering for large language modelscited by 72
  17. 2024LazyLLM: Dynamic Token Pruning for Efficient Long Context LLM Inferencecited by 70
  18. 2024Retrieval Augmented Generation (RAG) and Beyond: A Comprehensive Survey on How to Make your LLMs use External Data More Wiselycited by 66
  19. 2025ChunkKV: Semantic-Preserving KV Cache Compression for Efficient Long-Context LLM Inferencecited by 60
  20. 2024Multimodal Task Vectors Enable Many-Shot Multimodal In-Context Learningcited by 50
  21. 2025Token Reduction Should Go Beyond Efficiency in Generative Models -- From Vision, Language to Multimodalitycited by 50
  22. 2024DeciMamba: Exploring the Length Extrapolation Potential of Mambacited by 49
  23. 2024LongEmbed: Extending Embedding Models for Long Context Retrievalcited by 48
  24. 2024Efficient Prompting Methods for Large Language Models: A Surveycited by 47
  25. 2026SWE-Pruner: Self-Adaptive Context Pruning for Coding Agentscited by 44
  26. 2025LongCodeZip: Compress Long Context for Code Language Modelscited by 39
  27. 2025A Survey of Multimodal Retrieval-Augmented Generationcited by 37
  28. 2025PENCIL: Long Thoughts with Short Memorycited by 33
  29. 2024How Much Can RAG Help the Reasoning of LLM?cited by 31
  30. 2024GeoLLM-Engine: A Realistic Environment for Building Geospatial Copilotscited by 30
  31. 2024ALR2: A Retrieve-then-Reason Framework for Long-context Question Answeringcited by 26
  32. 2024Accelerating Inference of Retrieval-Augmented Generation via Sparse Context Selectioncited by 25
  33. 2024Compute Or Load KV Cache? Why Not Both?cited by 25
  34. 2025A Survey of Token Compression for Efficient Multimodal Large Language Modelscited by 25
  35. 2025SwiReasoning: Switch-Thinking in Latent and Explicit for Pareto-Superior Reasoning LLMscited by 22
  36. 2025Vision-centric Token Compression in Large Language Modelcited by 21
  37. 2025Text Generation Beyond Discrete Token Samplingcited by 21
  38. 2026Chain-of-Thought Compression Should Not Be Blind: V-Skip for Efficient Multimodal Reasoning via Dual-Path Anchoringcited by 21

50 most cited of 287 All 287 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2023Compress, Then Prompt: Improving Accuracy-Efficiency Trade-off of LLM Inference with Transferable Prompt
  2. 2025Dynamic Compressing Prompts for Efficient Inference of Large Language Models
  3. 2024LLMC: Benchmarking Large Language Model Quantization with a Versatile Compression Toolkit
  4. 2023A Survey on Model Compression for Large Language Models
  5. 2023Just CHOP: Embarrassingly Simple LLM Compression
  6. 2025Optimizing LLMs for Resource-Constrained Environments: A Survey of Model Compression Techniques
  7. 2025Cmprsr: Abstractive Token-Level Question-Agnostic Prompt Compressor
  8. 2024Enhancing and Accelerating Large Language Models via Instruction-Aware Contextual Compression