mlsanityData to 7 Oct 2026

Adapting Language Models to Compress Contexts

arXiv 2305.14788 ↗First posted 24 May 2023cs.CLcited by 350

Not on the map. Why?

Cited by 111 in Efficient LLM inference and serving, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Transformer-based language models (LMs) are powerful and widely-applicable tools, but their usefulness is constrained by a finite context window and the expensive computational cost of processing long text documents. We propose to adapt pre-trained LMs into AutoCompressors. These language models are capable of compressing long contexts into compact summary vectors, which are then accessible to the model as soft prompts. Summary vectors are trained with an unsupervised objective, whereby long documents are processed in segments, and summary vectors from all previous segments are used in language modeling. We fine-tune OPT and Llama-2 models on sequences of up to 30,720 tokens and show that AutoCompressors can utilize long contexts to improve perplexity. We evaluate AutoCompressors on in-context learning by compressing task demonstrations and find that summary vectors are good substitutes for plain-text demonstrations, increasing accuracy while reducing inference costs. Finally, we explore the benefits of pre-computing summary vectors for large corpora by applying summary vectors to retrievalaugmented language modeling and a passage re-ranking task. Overall, AutoCompressors emerge as a simple and inexpensive solution to extend the context window of LMs while speeding up inference over long contexts.

Key facts. Cited by 350 papers in the map (data through 7 Oct 2026). Builds on 40 of its 58 references that are in the map. First posted on arXiv 24 May 2023.

Builds on

40 of 58 references.

References · 40

  1. 2017Attention Is All You Needcited by 59,932
  2. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  3. 2020Language Models are Few-Shot Learnerscited by 32,199
  4. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  5. 2021Prefix-Tuning: Optimizing Continuous Prompts for Generationcited by 3,878
  6. 2020Dense Passage Retrieval for Open-Domain Question Answeringcited by 3,779
  7. 2021The Power of Scale for Parameter-Efficient Prompt Tuningcited by 3,769
  8. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,611
  9. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,464
  10. 2020The Pile: An 800GB Dataset of Diverse Text for Language Modelingcited by 2,346
  11. 2015Character-level Convolutional Networks for Text Classificationcited by 2,293
  12. 2019BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questionscited by 2,140
28 more
  1. 2020REALM: Retrieval-Augmented Language Model Pre-Trainingcited by 2,126
  2. 2019Transformer-XL: Attentive Language Models Beyond a Fixed-Length Contextcited by 1,830
  3. 2019SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systemscited by 1,718
  4. 2020Rethinking Attention with Performerscited by 1,584
  5. 2019Generating Long Sequences with Sparse Transformerscited by 1,523
  6. 2021Calibrate Before Use: Improving Few-Shot Performance of Language Modelscited by 1,371
  7. 2021Improving language models by retrieving from trillions of tokenscited by 1,278
  8. 2020Leveraging Passage Retrieval with Generative Models for Open Domain Question Answeringcited by 1,203
  9. 2021Unsupervised Dense Information Retrieval with Contrastive Learningcited by 1,203
  10. 2021Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolationcited by 1,085
  11. 2021A General Language Assistant as a Laboratory for Alignmentcited by 1,023
  12. 2016Training Deep Nets with Sublinear Memory Costcited by 825
  13. 2019Generalization through Memorization: Nearest Neighbor Language Modelscited by 788
  14. 2023REPLUG: Retrieval-Augmented Black-Box Language Modelscited by 786
  15. 2020Efficient Transformers: A Surveycited by 784
  16. 2019Compressive Transformers for Long-Range Sequence Modellingcited by 699
  17. 2023Learning to Compress Prompts with Gist Tokenscited by 379
  18. 2018WiC: the Word-in-Context Dataset for Evaluating Context-Sensitive Meaning Representationscited by 370
  19. 2021Factual Probing Is [MASK]: Learning vs. Learning to Recallcited by 291
  20. 2022Recurrent Memory Transformercited by 197
  21. 2022Mega: Moving Average Equipped Gated Attentioncited by 178
  22. 2023Scaling Down to Scale Up: A Guide to Parameter-Efficient Fine-Tuningcited by 171
  23. 2022Improving Passage Retrieval with Zero-Shot Question Generationcited by 164
  24. 2022Learning by Distilling Contextcited by 128
  25. 2022Training Language Models with Memory Augmentationcited by 112
  26. 2022Prompt Compression and Contrastive Conditioning for Controllability and Toxicity Reduction in Language Modelscited by 96
  27. 2023CoLT5: Faster Long-Range Transformers with Conditional Computationcited by 84
  28. 2022Linear Complexity Randomized Self-attention Mechanismcited by 31

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 20; 2024: 108; 2025: 89; 2026: 133.

350 papers.

Sort:

By area:

  1. 2024LongMemEval: Benchmarking Chat Assistants on Long-Term Interactive Memorycited by 656
  2. 2023LongLLMLingua: Accelerating and Enhancing LLMs in Long Context Scenarios via Prompt Compressioncited by 443
  3. 2023LoraHub: Efficient Cross-Task Generalization via Dynamic LoRA Compositioncited by 359
  4. 2024LLMLingua-2: Data Distillation for Efficient and Faithful Task-Agnostic Prompt Compressioncited by 343
  5. 2023LLMLingua: Compressing Prompts for Accelerated Inference of Large Language Modelscited by 279
  6. 2024Personal LLM Agents: Insights and Survey about the Capability, Efficiency and Securitycited by 257
  7. 2023RECOMP: Improving Retrieval-Augmented LMs with Compression and Selective Augmentationcited by 253
  8. 2025TokenSkip: Controllable Chain-of-Thought Compression in LLMscited by 246
  9. 2024BABILong: Testing the Limits of LLMs with Long Context Reasoning-in-a-Haystackcited by 211
  10. 2024A Survey on Efficient Inference for Large Language Modelscited by 197
  11. 2024Leave No Context Behind: Efficient Infinite Context Transformers with Infini-attentioncited by 194
  12. 2023In-context Autoencoder for Context Compression in a Large Language Modelcited by 190
38 more
  1. 2023Compressing Context to Enhance Inference Efficiency of Large Language Modelscited by 185
  2. 2024How to Train Long-Context Language Models (Effectively)cited by 141
  3. 2023Walking Down the Memory Maze: Beyond Context Limit through Interactive Readingcited by 129
  4. 2024C3oT: Generating Shorter Chain-of-Thought without Compromising Effectivenesscited by 117
  5. 2024mPLUG-DocOwl2: High-resolution Compressing for OCR-free Multi-page Document Understandingcited by 113
  6. 2024A Survey on Large Language Model-Based Game Agentscited by 106
  7. 2025LightThinker: Thinking Step-by-Step Compressioncited by 104
  8. 2023Towards Efficient Generative Large Language Model Serving: A Survey from Algorithms to Systemscited by 101
  9. 2024Long Context Compression with Activation Beaconcited by 101
  10. 2024HiAgent: Hierarchical Working Memory Management for Solving Long-Horizon Agent Tasks with Large Language Modelcited by 101
  11. 2024Keep the Cost Down: A Review on Methods to Optimize LLM' s KV-Cache Consumptioncited by 96
  12. 2024Long-Context Language Modeling with Parallel Context Encodingcited by 84
  13. 2024VoCo-LLaMA: Towards Vision Compression with Large Language Modelscited by 78
  14. 2025Lifelong Learning of Large Language Model based Agents: A Roadmapcited by 78
  15. 2024User-LLM: Efficient LLM Contextualization with User Embeddingscited by 74
  16. 2025Cartridges: Lightweight and general-purpose long context representations via self-studycited by 74
  17. 2023Unleashing the potential of prompt engineering for large language modelscited by 72
  18. 2025Efficient Reasoning with Hidden Thinkingcited by 72
  19. 2023Advancing Transformer Architecture in Long-Context Large Language Models: A Comprehensive Surveycited by 71
  20. 2024Retrieval-Augmented Generation for Natural Language Processing: A Surveycited by 70
  21. 2023EdgeMoE: Empowering Sparse Large Language Models on Mobile Devicescited by 65
  22. 2024Prompt Compression for Large Language Models: A Surveycited by 65
  23. 2024CompAct: Compressing Retrieved Documents Actively for Question Answeringcited by 64
  24. 2025ChunkKV: Semantic-Preserving KV Cache Compression for Efficient Long-Context LLM Inferencecited by 57
  25. 2024Learning to Compress Prompt in Natural Language Formatscited by 54
  26. 2024Sparser is Faster and Less is More: Efficient Sparse Attention for Long-Range Transformerscited by 51
  27. 2024Prompt Compression with Context-Aware Sentence Encoding for Fast and Improved LLM Inferencecited by 51
  28. 2024Multimodal Task Vectors Enable Many-Shot Multimodal In-Context Learningcited by 49
  29. 2025Token Reduction Should Go Beyond Efficiency in Generative Models -- From Vision, Language to Multimodalitycited by 49
  30. 2024Efficient Prompting Methods for Large Language Models: A Surveycited by 46
  31. 2023Extending Context Window of Large Language Models via Semantic Compressioncited by 45
  32. 2024DeciMamba: Exploring the Length Extrapolation Potential of Mambacited by 45
  33. 2024500xCompressor: Generalized Prompt Compression for Large Language Modelscited by 45
  34. 2024KV Cache Compression, But What Must We Give in Return? A Comprehensive Benchmark of Long Context Capable Approachescited by 43
  35. 2024NACL: A General and Effective KV Cache Eviction Framework for LLMs at Inference Timecited by 43
  36. 2023Discrete Prompt Compression with Reinforcement Learningcited by 42
  37. 2023Compressed Context Memory For Online Language Model Interactioncited by 42
  38. 2025Agentic Software Engineering: Foundational Pillars and a Research Roadmapcited by 40

50 most cited of 350 All 350 (JSON)