mlsanityData to 7 Oct 2026

Learning to Compress Prompts with Gist Tokens

arXiv 2304.08467 ↗First posted 17 Apr 2023cs.CLcited by 379

Not on the map. Why?

Cited by 118 in Efficient LLM inference and serving, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Prompting is the primary way to utilize the multitask capabilities of language models (LMs), but prompts occupy valuable space in the input context window, and repeatedly encoding the same prompt is computationally inefficient. Finetuning and distillation methods allow for specialization of LMs without prompting, but require retraining the model for each task. To avoid this trade-off entirely, we present gisting, which trains an LM to compress prompts into smaller sets of "gist" tokens which can be cached and reused for compute efficiency. Gist models can be trained with no additional cost over standard instruction finetuning by simply modifying Transformer attention masks to encourage prompt compression. On decoder (LLaMA-7B) and encoder-decoder (FLAN-T5-XXL) LMs, gisting enables up to 26x compression of prompts, resulting in up to 40% FLOPs reductions, 4.2% wall time speedups, and storage savings, all with minimal loss in output quality.

Key facts. Cited by 379 papers in the map (data through 7 Oct 2026). Builds on 36 of its 58 references that are in the map. First posted on arXiv 17 Apr 2023.

Builds on

36 of 58 references.

References · 36

  1. 2017Attention Is All You Needcited by 59,932
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
  4. 2022Training language models to follow instructions with human feedbackcited by 15,712
  5. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  6. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  7. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  8. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  9. 2015Distilling the Knowledge in a Neural Networkcited by 10,076
  10. 2019Parameter-Efficient Transfer Learning for NLPcited by 4,195
  11. 2021Prefix-Tuning: Optimizing Continuous Prompts for Generationcited by 3,878
  12. 2021The Power of Scale for Parameter-Efficient Prompt Tuningcited by 3,769
24 more
  1. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,747
  2. 2020Longformer: The Long-Document Transformercited by 3,362
  3. 2022Scaling Instruction-Finetuned Language Modelscited by 2,997
  4. 2022Training Compute-Optimal Large Language Modelscited by 2,927
  5. 2022Self-Instruct: Aligning Language Models with Self-Generated Instructionscited by 2,674
  6. 2019Transformer-XL: Attentive Language Models Beyond a Fixed-Length Contextcited by 1,830
  7. 2019Generating Long Sequences with Sparse Transformerscited by 1,523
  8. 2021Multitask Prompted Training Enables Zero-Shot Task Generalizationcited by 1,462
  9. 2021A General Language Assistant as a Laboratory for Alignmentcited by 1,023
  10. 2016HyperNetworkscited by 1,003
  11. 2022Super-NaturalInstructions: Generalization via Declarative Instructions on 1600+ NLP Taskscited by 858
  12. 2023ChatGPT Outperforms Crowd-Workers for Text-Annotation Taskscited by 849
  13. 2020Efficient Transformers: A Surveycited by 784
  14. 2019Compressive Transformers for Long-Range Sequence Modellingcited by 699
  15. 2022Efficiently Scaling Transformer Inferencecited by 529
  16. 2023Is ChatGPT a Good NLG Evaluator? A Preliminary Studycited by 420
  17. 2018Generating Wikipedia by Summarizing Long Sequencescited by 333
  18. 2022Memorizing Transformerscited by 219
  19. 2023Is ChatGPT better than Human Annotators? Potential and Limitations of ChatGPT in Explaining Implicit Hate Speechcited by 165
  20. 2022Learning by Distilling Contextcited by 128
  21. 2022Prompt Compression and Contrastive Conditioning for Controllability and Toxicity Reduction in Language Modelscited by 96
  22. 2021Poolingformer: Long Document Modeling with Pooling Attentioncited by 60
  23. 2022HyperTuning: Toward Adapting Large Language Models without Back-propagationcited by 45
  24. 2022Prompt Injection: Parameterization of Fixed Inputscited by 35

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 34; 2024: 111; 2025: 87; 2026: 147.

379 papers.

Sort:

By area:

  1. 2023H2O: Heavy-Hitter Oracle for Efficient Generative Inference of Large Language Modelscited by 805
  2. 2023Extending Context Window of Large Language Models via Positional Interpolationcited by 721
  3. 2024LongMemEval: Benchmarking Chat Assistants on Long-Term Interactive Memorycited by 656
  4. 2023Model Tells You What to Discard: Adaptive KV Cache Compression for LLMscited by 466
  5. 2023LongLLMLingua: Accelerating and Enhancing LLMs in Long Context Scenarios via Prompt Compressioncited by 443
  6. 2023Adapting Language Models to Compress Contextscited by 350
  7. 2024LLMLingua-2: Data Distillation for Efficient and Faithful Task-Agnostic Prompt Compressioncited by 343
  8. 2023Function Vectors in Large Language Modelscited by 292
  9. 2024Quiet-STaR: Language Models Can Teach Themselves to Think Before Speakingcited by 292
  10. 2023LLMLingua: Compressing Prompts for Accelerated Inference of Large Language Modelscited by 279
  11. 2023RECOMP: Improving Retrieval-Augmented LMs with Compression and Selective Augmentationcited by 253
  12. 2023The Unlocking Spell on Base LLMs: Rethinking Alignment via In-Context Learningcited by 251
38 more
  1. 2025TokenSkip: Controllable Chain-of-Thought Compression in LLMscited by 246
  2. 2023HyperDreamBooth: HyperNetworks for Fast Personalization of Text-to-Image Modelscited by 224
  3. 2023Landmark Attention: Random-Access Infinite Context Length for Transformerscited by 206
  4. 2023CacheGen: KV Cache Compression and Streaming for Fast Large Language Model Servingcited by 199
  5. 2024A Survey on Efficient Inference for Large Language Modelscited by 197
  6. 2024Leave No Context Behind: Efficient Infinite Context Transformers with Infini-attentioncited by 194
  7. 2023In-context Autoencoder for Context Compression in a Large Language Modelcited by 190
  8. 2023Compressing Context to Enhance Inference Efficiency of Large Language Modelscited by 185
  9. 2023Efficient Large Language Models: A Surveycited by 183
  10. 2024From Decoding to Meta-Generation: Inference-time Algorithms for Large Language Modelscited by 136
  11. 2024A Survey on Large Language Model Acceleration based on KV Cache Managementcited by 125
  12. 2024Keyformer: KV Cache Reduction through Key Tokens Selection for Efficient Generative Inferencecited by 124
  13. 2024xRAG: Extreme Context Compression for Retrieval-augmented Generation with One Tokencited by 123
  14. 2024C3oT: Generating Shorter Chain-of-Thought without Compromising Effectivenesscited by 117
  15. 2024Dynamic Memory Compression: Retrofitting LLMs for Accelerated Inferencecited by 115
  16. 2024A Survey on Large Language Model-Based Game Agentscited by 106
  17. 2025LightThinker: Thinking Step-by-Step Compressioncited by 104
  18. 2023Towards Efficient Generative Large Language Model Serving: A Survey from Algorithms to Systemscited by 101
  19. 2024Long Context Compression with Activation Beaconcited by 101
  20. 2024Keep the Cost Down: A Review on Methods to Optimize LLM' s KV-Cache Consumptioncited by 96
  21. 2024VoCo-LLaMA: Towards Vision Compression with Large Language Modelscited by 78
  22. 2025Lifelong Learning of Large Language Model based Agents: A Roadmapcited by 78
  23. 2024User-LLM: Efficient LLM Contextualization with User Embeddingscited by 74
  24. 2025Cartridges: Lightweight and general-purpose long context representations via self-studycited by 74
  25. 2023Advancing Transformer Architecture in Long-Context Large Language Models: A Comprehensive Surveycited by 71
  26. 2024Prompt Compression for Large Language Models: A Surveycited by 65
  27. 2024CompAct: Compressing Retrieved Documents Actively for Question Answeringcited by 64
  28. 2023Guiding Language Model Reasoning with Planning Tokenscited by 58
  29. 2023Large Language Models are Visual Reasoning Coordinatorscited by 58
  30. 2024Layer-Condensed KV Cache for Efficient Inference of Large Language Modelscited by 58
  31. 2024DoraemonGPT: Toward Understanding Dynamic Scenes with Large Language Models (Exemplified as A Video Agent)cited by 56
  32. 2024Learning to Compress Prompt in Natural Language Formatscited by 54
  33. 2024LoRC: Low-Rank Compression for LLMs KV Cache with a Progressive Compression Strategycited by 50
  34. 2025Text-to-LoRA: Instant Transformer Adaptioncited by 50
  35. 2024Multimodal Task Vectors Enable Many-Shot Multimodal In-Context Learningcited by 49
  36. 2025Token Reduction Should Go Beyond Efficiency in Generative Models -- From Vision, Language to Multimodalitycited by 49
  37. 2023EcoAssistant: Using LLM Assistant More Affordably and Accuratelycited by 47
  38. 2024Efficient Prompting Methods for Large Language Models: A Surveycited by 46

50 most cited of 379 All 379 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025Task-agnostic Prompt Compression with Context-aware Sentence Embedding and Reward-guided Task Descriptor
  2. 2026Prompt Minimization: Reducing Input Redundancy Without Sacrificing Output Fidelity
  3. 2024Style-Compress: An LLM-Based Prompt Compression Framework Considering Task-Specific Styles
  4. 2025Learning a Single Token to Replace Long System Prompts in LLMs
  5. 2025An Empirical Study on Prompt Compression for Large Language Models
  6. 2024LanguaShrink: Reducing Token Overhead with Psycholinguistics
  7. 2026Prompt Compression in the Wild: Measuring Latency, Rate Adherence, and Quality for Faster LLM Inference
  8. 2023Compresso: Structured Pruning with Collaborative Prompting Learns Compact Large Language Models