mlsanityData to 14 Oct 2026

Long Context In-Context Compression by Getting to the Gist of Gisting

arXiv 2504.08934 ↗First posted 11 Apr 2025cs.LG cs.AIcited by 13

On the map: Efficient LLM inference and serving → Prompt and context compression

Abstract (arXiv)

Long context processing is critical for the adoption of LLMs, but existing methods often introduce architectural complexity that hinders their practical adoption. Gisting, an in-context compression method with no architectural modification to the decoder transformer, is a promising approach due to its simplicity and compatibility with existing frameworks. While effective for short instructions, we demonstrate that gisting struggles with longer contexts, with significant performance drops even at minimal compression rates. Surprisingly, a simple average pooling baseline consistently outperforms gisting. We analyze the limitations of gisting, including information flow interruptions, capacity limitations and the inability to restrict its attention to subsets of the context. Motivated by theoretical insights into the performance gap between gisting and average pooling, and supported by extensive experimentation, we propose GistPool, a new in-context compression method. GistPool preserves the simplicity of gisting, while significantly boosting its performance on long context compression tasks.

Key facts. Cited by 13 papers in the map (data through 14 Oct 2026). Placed in Prompt and context compression, one of 12 directions in Efficient LLM inference and serving. Builds on 34 of its 46 references that are in the map. First posted on arXiv 11 Apr 2025.

Builds on

34 of 46 references.

Closest ancestors · 6

Same direction, newest first

  1. 2024A Silver Bullet or a Compromise for Full Attention? A Comprehensive Study of Gist Token-based Context Compression23 Dec
  2. 2024Retaining Key Information under High Compression Ratios: Query-Guided Compressor for LLMs4 Jun
  3. 2024LLMLingua-2: Data Distillation for Efficient and Faithful Task-Agnostic Prompt Compression19 Mar
  4. 2023LongLLMLingua: Accelerating and Enhancing LLMs in Long Context Scenarios via Prompt Compression10 Oct
  5. 2023Discrete Prompt Compression with Reinforcement Learning17 Aug
  6. 2022Prompt Compression and Contrastive Conditioning for Controllability and Toxicity Reduction in Language Models6 Oct

Other references · 28

  1. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
  2. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,297
  3. 2016SQuAD: 100,000+ Questions for Machine Comprehension of Textcited by 4,148
  4. 2021Prefix-Tuning: Optimizing Continuous Prompts for Generationcited by 3,915
  5. 2023Lost in the Middle: How Language Models Use Long Contextscited by 3,455
  6. 2020Longformer: The Long-Document Transformercited by 3,413
  7. 2022Self-Instruct: Aligning Language Models with Self-Generated Instructionscited by 2,724
  8. 2023Efficient Streaming Language Models with Attention Sinkscited by 2,186
  9. 2020Big Bird: Transformers for Longer Sequencescited by 1,684
  10. 2017Deep Setscited by 1,421
  11. 2018Adafactor: Adaptive Learning Rates with Sublinear Memory Costcited by 905
  12. 2019DROP: A Reading Comprehension Benchmark Requiring Discrete Reasoning Over Paragraphscited by 903
16 more
  1. 2017RACE: Large-scale ReAding Comprehension Dataset From Examinationscited by 861
  2. 2022Transformers learn in-context by gradient descentcited by 732
  3. 2022What learning algorithm is in-context learning? Investigations with linear modelscited by 684
  4. 2024Parameter-Efficient Fine-Tuning for Large Models: A Comprehensive Surveycited by 656
  5. 2017The NarrativeQA Reading Comprehension Challengecited by 597
  6. 2023Learning to Compress Prompts with Gist Tokenscited by 390
  7. 2023Adapting Language Models to Compress Contextscited by 356
  8. 2016Towards a Neural Statisticiancited by 121
  9. 2024Transformers need glasses! Information over-squashing in language taskscited by 91
  10. 2024Softmax is not Enough (for Sharp Size Generalisation)cited by 46
  11. 2023Compressed Context Memory For Online Language Model Interactioncited by 43
  12. 2023PRCA: Fitting Black-Box Large Language Models for Retrieval Question Answering via Pluggable Reward-Driven Contextual Adaptercited by 38
  13. 2024Prompting a Pretrained Transformer Can Be a Universal Approximatorcited by 25
  14. 2024LoCoCo: Dropping In Convolutions for Long Context Compressioncited by 19
  15. 2024Say More with Less: Understanding Prompt Learning Behaviors through Gist Compressioncited by 8
  16. 2024SelfCP: Compressing Over-Limit Prompt via the Frozen Large Language Model Itselfcited by 8

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 3; 2026: 10.

13 papers.

Sort:

By area:

  1. 2026COMI: Coarse-to-fine Context Compression via Marginal Information Gaincited by 13
  2. 2026Read As Human: Compressing Context via Parallelizable Close Reading and Skimmingcited by 10
  3. 2026Simplified Sparse Attention via Gist Tokenscited by 6
  4. 2025CompLLM: Compression for Long Context Q&Acited by 5
  5. 2025See, Think, Act: Online Shopper Behavior Simulation with VLM Agentscited by 5
  6. 2026Compositional Steering of Large Language Models with Steering Tokenscited by 5
  7. 2025Towards Efficient Agents: A Co-Design of Inference Architecture and Systemcited by 4
  8. 2026MARCH: Scaling Recurrent Memory with Content-Routed State Anchorscited by 2
  9. 2026Cartridges++: KV Cache Compression without Off-Context Derailmentcited by 2
  10. 2026Safin-1: Safety from Within through Memory-Native State Evolutioncited by 1
  11. 2026ReSum: Synergizing LLM Reasoning and Summarization with Reinforcement Learningcited by 0
  12. 2026Repurposing Pre-trained LLMs as High Fidelity Continuous Text Autoencoderscited by 0
1 more
  1. 2026WIRE: Weight Space Implicit Representation Encoding of Frozen Video LLMscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Sentence-Anchored Gist Compression for Long-Context LLMs
  2. 2024Characterizing Prompt Compression Methods for Long Context Inference
  3. 2024Long Context Compression with Activation Beacon
  4. 2025ILRe: Intermediate Layer Retrieval for Context Compression in Causal Language Models
  5. 2024Enhancing and Accelerating Large Language Models via Instruction-Aware Contextual Compression
  6. 2024Recurrent Context Compression: Efficiently Expanding the Context Window of LLM
  7. 2026Jet-Long: Efficient Long-Context Extension with Dynamic Bifocal RoPE
  8. 2026Fix the Structural Bottleneck: Context Compression via Explicit Information Transmission