mlsanityData to 14 Oct 2026

Sentence-Anchored Gist Compression for Long-Context LLMs

arXiv 2511.08128 ↗First posted 11 Nov 2025cs.CLcited by 3

On the map: Efficient LLM inference and serving → Prompt and context compression

Abstract (arXiv)

This work investigates context compression for Large Language Models (LLMs) using learned compression tokens to reduce the memory and computational demands of processing long sequences. We demonstrate that pre-trained LLMs can be fine-tuned to compress their context by factors of 2x to 8x without significant performance degradation, as evaluated on both short-context and long-context benchmarks. Furthermore, in experiments on a 3-billion-parameter LLaMA model, our method achieves results on par with alternative compression techniques while attaining higher compression ratios.

Key facts. Cited by 3 papers in the map (data through 14 Oct 2026). Placed in Prompt and context compression, one of 12 directions in Efficient LLM inference and serving. Builds on 18 of its 31 references that are in the map. First posted on arXiv 11 Nov 2025.

Builds on

18 of 31 references.

Closest ancestors · 2

Same direction, newest first

  1. 2024A Silver Bullet or a Compromise for Full Attention? A Comprehensive Study of Gist Token-based Context Compression23 Dec
  2. 2024Long Context Compression with Activation Beacon7 Jan

Other references · 16

  1. 2020Measuring Massive Multitask Language Understandingcited by 7,963
  2. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,886
  3. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,100
  4. 2023Efficient Streaming Language Models with Attention Sinkscited by 2,186
  5. 2021Scaling Vision Transformerscited by 999
  6. 2024SnapKV: LLM Knows What You are Looking for Before Generationcited by 880
  7. 2019Compressive Transformers for Long-Range Sequence Modellingcited by 726
  8. 2023Model Tells You What to Discard: Adaptive KV Cache Compression for LLMscited by 474
  9. 2025Native Sparse Attention: Hardware-Aligned and Natively Trainable Sparse Attentioncited by 407
  10. 2023Learning to Compress Prompts with Gist Tokenscited by 390
  11. 2023Adapting Language Models to Compress Contextscited by 356
  12. 2024HELMET: How to Evaluate Long-Context Language Models Effectively and Thoroughlycited by 125
4 more
  1. 2024Model Tells You Where to Merge: Adaptive KV Cache Merging for LLMs on Long-Context Taskscited by 96
  2. 2024SepLLM: Accelerate Large Language Models by Compressing One Segment into One Separatorcited by 41
  3. 2025Cramming 1568 Tokens into a Single Vector and Back Again: Exploring the Limits of Embedding Space Capacitycited by 28
  4. 2025Adaptive Computation Pruning for the Forgetting Transformercited by 5

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 1; 2026: 2.

3 papers.

  1. 2025Memory in the Age of AI Agentscited by 303
  2. 2026SAM: State-Adaptive Memory for Long-Horizon Reasoning Agentcited by 10
  3. 2026Compositional Steering of Large Language Models with Steering Tokenscited by 5

Similar, not cited

Close in wording, no citation either way

  1. 2024Recurrent Context Compression: Efficiently Expanding the Context Window of LLM
  2. 2023A Survey on Model Compression for Large Language Models
  3. 2025Beyond Hard and Soft: Hybrid Context Compression for Balancing Local and Global Information Retention
  4. 2024Enhancing and Accelerating Large Language Models via Instruction-Aware Contextual Compression
  5. 2024KV Cache Compression, But What Must We Give in Return? A Comprehensive Benchmark of Long Context Capable Approaches
  6. 2023Compressing LLMs: The Truth is Rarely Pure and Never Simple
  7. 2026Compressing Long Context into Answer-Aligned Memory Embeddings for LLM Inference
  8. 2025Semantic Retention and Extreme Compression in LLMs: Can We Have Both?