mlsanityData to 7 Oct 2026

Prompt Compression and Contrastive Conditioning for Controllability and Toxicity Reduction in Language Models

arXiv 2210.03162 ↗First posted 6 Oct 2022cs.CL cs.AI cs.LGcited by 96

On the map: Efficient LLM inference and serving → Prompt and context compression

Abstract (arXiv)

We explore the idea of compressing the prompts used to condition language models, and show that compressed prompts can retain a substantive amount of information about the original prompt. For severely compressed prompts, while fine-grained information is lost, abstract information and general sentiments can be retained with surprisingly few parameters, which can be useful in the context of decode-time algorithms for controllability and toxicity reduction. We explore contrastive conditioning to steer language model generation towards desirable text and away from undesirable text, and find that some complex prompts can be effectively compressed into a single token to guide generation. We also show that compressed prompts are largely compositional, and can be constructed such that they can be used to control independent aspects of generated text.

Key facts. Cited by 96 papers in the map (data through 7 Oct 2026). Placed in Prompt and context compression, one of 12 directions in Efficient LLM inference and serving. Builds on 22 of its 39 references that are in the map. First posted on arXiv 6 Oct 2022.

Builds on

22 of 39 references.

References · 22

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2020Language Models are Few-Shot Learnerscited by 32,199
  4. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  5. 2017Proximal Policy Optimization Algorithmscited by 14,441
  6. 2021The Power of Scale for Parameter-Efficient Prompt Tuningcited by 3,769
  7. 2019The Curious Case of Neural Text Degenerationcited by 2,881
  8. 2020Learning to summarize from human feedbackcited by 2,562
  9. 2020The Pile: An 800GB Dataset of Diverse Text for Language Modelingcited by 2,346
  10. 2019Transformer-XL: Attentive Language Models Beyond a Fixed-Length Contextcited by 1,830
  11. 2020Knowledge Distillation: A Surveycited by 1,578
  12. 2020RealToxicityPrompts: Evaluating Neural Toxic Degeneration in Language Modelscited by 1,235
10 more
  1. 2016Semantics derived automatically from language corpora contain human-like biasescited by 894
  2. 2019DialoGPT: Large-Scale Generative Pre-training for Conversational Response Generationcited by 842
  3. 2019Plug and Play Language Models: A Simple Approach to Controlled Text Generationcited by 793
  4. 2019CTRL: A Conditional Transformer Language Model for Controllable Generationcited by 754
  5. 2021Prompt Programming for Large Language Models: Beyond the Few-Shot Paradigmcited by 599
  6. 2020Towards a Human-like Open-Domain Chatbotcited by 467
  7. 2021DExperts: Decoding-Time Controlled Text Generation with Experts and Anti-Expertscited by 434
  8. 2020GeDi: Generative Discriminator Guided Sequence Generationcited by 377
  9. 2017Word Embeddings Quantify 100 Years of Gender and Ethnic Stereotypescited by 251
  10. 2022Quark: Controllable Text Generation with Reinforced Unlearningcited by 236

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 17; 2024: 33; 2025: 35; 2026: 11.

96 papers.

Sort:

By area:

  1. 2023Learning to Compress Prompts with Gist Tokenscited by 379
  2. 2023Adapting Language Models to Compress Contextscited by 350
  3. 2024LLMLingua-2: Data Distillation for Efficient and Faithful Task-Agnostic Prompt Compressioncited by 343
  4. 2023LLMLingua: Compressing Prompts for Accelerated Inference of Large Language Modelscited by 279
  5. 2023RECOMP: Improving Retrieval-Augmented LMs with Compression and Selective Augmentationcited by 253
  6. 2025TokenSkip: Controllable Chain-of-Thought Compression in LLMscited by 246
  7. 2024A Survey on Efficient Inference for Large Language Modelscited by 197
  8. 2023In-context Autoencoder for Context Compression in a Large Language Modelcited by 190
  9. 2023ReWOO: Decoupling Reasoning from Observations for Efficient Augmented Language Modelscited by 174
  10. 2024EASYTOOL: Enhancing LLM-based Agents with Concise Tool Instructioncited by 153
  11. 2024C3oT: Generating Shorter Chain-of-Thought without Compromising Effectivenesscited by 117
  12. 2024VoCo-LLaMA: Towards Vision Compression with Large Language Modelscited by 78
38 more
  1. 2023Advancing Transformer Architecture in Long-Context Large Language Models: A Comprehensive Surveycited by 71
  2. 2024Prompt Compression for Large Language Models: A Surveycited by 65
  3. 2025ChunkKV: Semantic-Preserving KV Cache Compression for Efficient Long-Context LLM Inferencecited by 57
  4. 2024Learning to Compress Prompt in Natural Language Formatscited by 54
  5. 2025Token Reduction Should Go Beyond Efficiency in Generative Models -- From Vision, Language to Multimodalitycited by 49
  6. 2024Efficient Prompting Methods for Large Language Models: A Surveycited by 46
  7. 2023Extending Context Window of Large Language Models via Semantic Compressioncited by 45
  8. 2024DeciMamba: Exploring the Length Extrapolation Potential of Mambacited by 45
  9. 2024500xCompressor: Generalized Prompt Compression for Large Language Modelscited by 45
  10. 2024KV Cache Compression, But What Must We Give in Return? A Comprehensive Benchmark of Long Context Capable Approachescited by 43
  11. 2023Discrete Prompt Compression with Reinforcement Learningcited by 42
  12. 2023Compressed Context Memory For Online Language Model Interactioncited by 42
  13. 2025Agentic Software Engineering: Foundational Pillars and a Research Roadmapcited by 40
  14. 2025A Survey of Multimodal Retrieval-Augmented Generationcited by 37
  15. 2024Online Adaptation of Language Models with a Memory of Amortized Contextscited by 35
  16. 2023DePT: Decomposed Prompt Tuning for Parameter-Efficient Fine-tuningcited by 34
  17. 2024Symbolic Prompt Program Search: A Structure-Aware Approach to Efficient Compile-Time Prompt Optimizationcited by 29
  18. 2023Fewer is More: Boosting LLM Reasoning with Reinforced Context Pruningcited by 26
  19. 2023Dodo: Dynamic Contextual Compression for Decoder-only LMscited by 23
  20. 2025Vision-centric Token Compression in Large Language Modelcited by 21
  21. 2025PISCO: Pretty Simple Compression for Retrieval-Augmented Generationcited by 20
  22. 2025SwiReasoning: Switch-Thinking in Latent and Explicit for Pareto-Superior Reasoning LLMscited by 20
  23. 2024Prompt-SAW: Leveraging Relation-Aware Graphs for Textual Prompt Compressioncited by 18
  24. 2024Fundamental Limits of Prompt Compression: A Rate-Distortion Framework for Black-Box Language Modelscited by 18
  25. 2024EXIT: Context-Aware Extractive Compression for Enhancing Retrieval-Augmented Generationcited by 17
  26. 2024Compressing Long Context for Enhancing RAG with AMR-based Concept Distillationcited by 16
  27. 2025Semantic Integrity Matters: Benchmarking and Preserving High-Density Reasoning in KV Cache Compressioncited by 16
  28. 2025InfiniteICL: Breaking the Limit of Context Window Size via Long Short-term Memory Transformationcited by 16
  29. 2025Autoencoding-Free Context Compression for LLMs via Contextual Semantic Anchorscited by 14
  30. 2024PromptIntern: Saving Inference Costs by Internalizing Recurrent Prompt during Large Language Model Fine-tuningcited by 13
  31. 2025Understanding and Improving Information Preservation in Prompt Compression for LLMscited by 13
  32. 2024In-Context Former: Lightning-fast Compressing Context for Large Language Modelcited by 12
  33. 2025Long Context In-Context Compression by Getting to the Gist of Gistingcited by 12
  34. 2023AI Text-to-Behavior: A Study In Steerabilitycited by 11
  35. 2024RSA-Control: A Pragmatics-Grounded Lightweight Controllable Text Generation Frameworkcited by 11
  36. 2025Not All Errors Are Created Equal: ASCoT Addresses Late-Stage Fragility in Efficient LLM Reasoningcited by 11
  37. 2026Latent Context Compilation: Distilling Long Context into Compact Portable Memorycited by 11
  38. 2023Optimizing Retrieval-augmented Reader Models via Token Eliminationcited by 10

50 most cited of 96 All 96 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2026Prompt Minimization: Reducing Input Redundancy Without Sacrificing Output Fidelity
  2. 2026Lexical Prompt Compression for Large Language Models: A Training-Free, Deterministic Pipeline with Empirical Pareto Analysis Across Eleven Task Categories
  3. 2026CAVEWOMAN: How Large Language Models Behave Under Linguistic Input and Output Compression
  4. 2026Every Time I Hire a Linguist, Inference Costs Go Down: On Linguistic Rules as Effective Prompt Compressors
  5. 2025An Empirical Study on Prompt Compression for Large Language Models
  6. 2025FrugalPrompt: Reducing Contextual Overhead in Large Language Models via Token Attribution
  7. 2025Task-agnostic Prompt Compression with Context-aware Sentence Embedding and Reward-guided Task Descriptor
  8. 2026Prompt Compression via Activation Aggregation