mlsanityData to 7 Oct 2026

A Silver Bullet or a Compromise for Full Attention? A Comprehensive Study of Gist Token-based Context Compression

arXiv 2412.17483 ↗First posted 23 Dec 2024cs.CLcited by 15

On the map: Efficient LLM inference and serving → Prompt and context compression

Abstract (arXiv)

In this work, we provide a thorough investigation of gist-based context compression methods to improve long-context processing in large language models. We focus on two key questions: (1) How well can these methods replace full attention models? and (2) What potential failure patterns arise due to compression? Through extensive experiments, we show that while gist-based compression can achieve near-lossless performance on tasks like retrieval-augmented generation and long-document QA, it faces challenges in tasks like synthetic recall. Furthermore, we identify three key failure patterns: lost by the boundary, lost if surprise, and lost along the way. To mitigate these issues, we propose two effective strategies: fine-grained autoencoding, which enhances the reconstruction of original token information, and segment-wise token importance estimation, which adjusts optimization based on token dependencies. Our work provides valuable insights into the understanding of gist token-based context compression and offers practical strategies for improving compression capabilities.

Key facts. Cited by 15 papers in the map (data through 7 Oct 2026). Placed in Prompt and context compression, one of 12 directions in Efficient LLM inference and serving. Builds on 44 of its 60 references that are in the map. First posted on arXiv 23 Dec 2024.

Builds on

44 of 60 references.

Closest ancestors · 2

Same direction, newest first

  1. 2024Long Context Compression with Activation Beacon7 Jan
  2. 2023LongLLMLingua: Accelerating and Enhancing LLMs in Long Context Scenarios via Prompt Compression10 Oct

Other references · 42

  1. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  2. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,018
  3. 2023Let's Verify Step by Stepcited by 3,598
  4. 2020Longformer: The Long-Document Transformercited by 3,362
  5. 2023Retrieval-Augmented Generation for Large Language Models: A Surveycited by 2,185
  6. 2024Qwen2 Technical Reportcited by 2,076
  7. 2022Challenging BIG-Bench Tasks and Whether Chain-of-Thought Can Solve Themcited by 1,767
  8. 2024MMLU-Pro: A More Robust and Challenging Multi-Task Language Understanding Benchmarkcited by 1,758
  9. 2020Reformer: The Efficient Transformercited by 1,658
  10. 2023LongBench: A Bilingual, Multitask Benchmark for Long Context Understandingcited by 1,378
  11. 2023GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpointscited by 1,311
  12. 2024RULER: What's the Real Context Size of Your Long-Context Language Models?cited by 1,270
30 more
  1. 2024DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Modelcited by 1,093
  2. 2024SnapKV: LLM Knows What You are Looking for Before Generationcited by 848
  3. 2020Efficient Transformers: A Surveycited by 784
  4. 2023Extending Context Window of Large Language Models via Positional Interpolationcited by 721
  5. 2019Fast Transformer Decoding: One Write-Head is All You Needcited by 700
  6. 2019Compressive Transformers for Long-Range Sequence Modellingcited by 699
  7. 2024A Survey on the Memory Mechanism of Large Language Model based Agentscited by 622
  8. 2024KIVI: A Tuning-Free Asymmetric 2bit Quantization for KV Cachecited by 604
  9. 2023Model Tells You What to Discard: Adaptive KV Cache Compression for LLMscited by 466
  10. 2023Scissorhands: Exploiting the Persistence of Importance Hypothesis for LLM KV Cache Compression at Test Timecited by 428
  11. 2023Learning to Compress Prompts with Gist Tokenscited by 379
  12. 2024MInference 1.0: Accelerating Pre-filling for Long-Context LLMs via Dynamic Sparse Attentioncited by 375
  13. 2024∞Bench: Extending Long Context Evaluation Beyond 100K Tokenscited by 360
  14. 2023Adapting Language Models to Compress Contextscited by 350
  15. 2023RepoBench: Benchmarking Repository-Level Code Auto-Completion Systemscited by 346
  16. 2023Large Language Models for Information Retrieval: A Surveycited by 331
  17. 2024DuoAttention: Efficient Long-Context LLM Inference with Retrieval and Streaming Headscited by 283
  18. 2024Ada-KV: Optimizing KV Cache Eviction by Adaptive Budget Allocation for Efficient LLM Inferencecited by 230
  19. 2021BookSum: A Collection of Datasets for Long-form Narrative Summarizationcited by 184
  20. 2024Data Engineering for Scaling Language Models to 128K Contextcited by 183
  21. 2024How to Train Long-Context Language Models (Effectively)cited by 141
  22. 2024You Only Cache Once: Decoder-Decoder Architectures for Language Modelscited by 135
  23. 2024Reducing Transformer Key-Value Cache Size with Cross-Layer Attentioncited by 122
  24. 2024HELMET: How to Evaluate Long-Context Language Models Effectively and Thoroughlycited by 119
  25. 2024A Simple and Effective L2 Norm-Based Strategy for KV Cache Compressioncited by 84
  26. 2024What is Wrong with Perplexity for Long-context Language Modeling?cited by 68
  27. 2024Layer-Condensed KV Cache for Efficient Inference of Large Language Modelscited by 58
  28. 2023Dodo: Dynamic Contextual Compression for Decoder-only LMscited by 23
  29. 2023Nugget: Neural Agglomerative Embeddings of Textcited by 19
  30. 2022Socialformer: Social Network Inspired Long Document Modeling for Document Rankingcited by 2

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 7; 2026: 8.

15 papers.

Sort:

By area:

  1. 2025Long Context In-Context Compression by Getting to the Gist of Gistingcited by 12
  2. 2026Latent Context Compilation: Distilling Long Context into Compact Portable Memorycited by 11
  3. 2025GEM: Empowering LLM for both Embedding Generation and Language Understandingcited by 8
  4. 2025UniGist: Towards General and Hardware-aligned Sequence-level Long Context Compressioncited by 8
  5. 2025Beyond Hard and Soft: Hybrid Context Compression for Balancing Local and Global Information Retentioncited by 7
  6. 2025AdmTree: Compressing Lengthy Context with Adaptive Semantic Treescited by 7
  7. 2026What to Keep, What to Forget: A Rate--Distortion View of Memory Compaction in LLMs and Agentscited by 4
  8. 2025Sentence-Anchored Gist Compression for Long-Context LLMscited by 3
  9. 2025InComeS: Integrating Compression and Selection Mechanisms into LLMs for Efficient Model Editingcited by 2
  10. 2026Fix the Structural Bottleneck: Context Compression via Explicit Information Transmissioncited by 2
  11. 2026Efficient Sequential Recommendation for Long Term User Interest Via Personalizationcited by 1
  12. 2026ArcAligner: Adaptive Recursive Aligner for Compressed Context Embeddings in RAGcited by 1
3 more
  1. 2026Cartridges++: KV Cache Compression without Off-Context Derailmentcited by 1
  2. 2026Relevant but Incomplete: Referential Dangling as a Paradigm-Level Failure Mode in Hard Prompt Compressioncited by 0
  3. 2026LifeFuse-Mem: Lifecycle-Aware State Fusion Against Temporary Overwriting for Long-Term Memorycited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Characterizing Prompt Compression Methods for Long Context Inference
  2. 2026The Sleeping Agent: What Gist-Based Context Compression Loses and Why
  3. 2026Simplified Sparse Attention via Gist Tokens
  4. 2025AttnComp: Attention-Guided Adaptive Context Compression for Retrieval-Augmented Generation
  5. 2024Recurrent Context Compression: Efficiently Expanding the Context Window of LLM
  6. 2024In-Context Former: Lightning-fast Compressing Context for Large Language Model
  7. 2025CCF: A Context Compression Framework for Efficient Long-Sequence Language Modeling
  8. 2024Enhancing and Accelerating Large Language Models via Instruction-Aware Contextual Compression