mlsanityData to 14 Oct 2026

UniGist: Towards General and Hardware-aligned Sequence-level Long Context Compression

arXiv 2509.15763 ↗First posted 19 Sep 2025cs.CLcited by 9

On the map: Efficient LLM inference and serving → KV cache optimization

Abstract (arXiv)

Large language models are increasingly capable of handling long-context inputs, but the memory overhead of key-value (KV) cache remains a major bottleneck for general-purpose deployment. While various compression strategies have been explored, sequence-level compression, which drops the full KV caches for certain tokens, is particularly challenging as it can lead to the loss of important contextual information. To address this, we introduce UniGist, a sequence-level long-context compression framework that efficiently preserves context information by replacing raw tokens with special compression tokens (gists) in a fine-grained manner. We adopt a chunk-free training strategy and design an efficient kernel with a gist shift trick, enabling optimized GPU training. Our scheme also supports flexible inference by allowing the actual removal of compressed tokens, resulting in real-time memory savings. Experiments across multiple long-context tasks demonstrate that UniGist significantly improves compression quality, with especially strong performance in detail-recalling tasks and long-range dependency modeling.

Key facts. Cited by 9 papers in the map (data through 14 Oct 2026). Placed in KV cache optimization, one of 12 directions in Efficient LLM inference and serving. Builds on 47 of its 56 references that are in the map. First posted on arXiv 19 Sep 2025.

Builds on

47 of 56 references.

Closest ancestors · 9

Same direction, newest first

  1. 2024TokenSelect: Efficient Long-Context Inference and Length Extrapolation for LLMs via Dynamic Token-Level KV Cache Selection5 Nov
  2. 2024Not All Heads Matter: A Head-Level KV Cache Compression Method with Integrated Retrieval and Reasoning25 Oct
  3. 2024Ada-KV: Optimizing KV Cache Eviction by Adaptive Budget Allocation for Efficient LLM Inference16 Jul
  4. 2024Model Tells You Where to Merge: Adaptive KV Cache Merging for LLMs on Long-Context Tasks11 Jul
  5. 2024PyramidKV: Dynamic KV Cache Compression based on Pyramidal Information Funneling4 Jun
  6. 2024Reducing Transformer Key-Value Cache Size with Cross-Layer Attention21 May
  7. 2024PyramidInfer: Pyramid KV Cache Compression for High-throughput LLM Inference21 May
  8. 2024SnapKV: LLM Knows What You are Looking for Before Generation22 Apr
  9. 2023H2O: Heavy-Hitter Oracle for Efficient Generative Inference of Large Language Models24 Jun

Other references · 38

  1. 2017Attention Is All You Needcited by 60,361
  2. 2023GPT-4 Technical Reportcited by 18,396
  3. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  4. 2023Mamba: Linear-Time Sequence Modeling with Selective State Spacescited by 4,718
  5. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,100
  6. 2020Longformer: The Long-Document Transformercited by 3,413
  7. 2023FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioningcited by 2,598
  8. 2023A Survey of Large Language Modelscited by 2,576
  9. 2023Retrieval-Augmented Generation for Large Language Models: A Surveycited by 2,212
  10. 2023Efficient Streaming Language Models with Attention Sinkscited by 2,186
  11. 2024MMLU-Pro: A More Robust and Challenging Multi-Task Language Understanding Benchmarkcited by 1,848
  12. 2020Big Bird: Transformers for Longer Sequencescited by 1,684
26 more
  1. 2023GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpointscited by 1,346
  2. 2024RULER: What's the Real Context Size of Your Long-Context Language Models?cited by 1,346
  3. 2024DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Modelcited by 1,135
  4. 2019Fast Transformer Decoding: One Write-Head is All You Needcited by 717
  5. 2024KIVI: A Tuning-Free Asymmetric 2bit Quantization for KV Cachecited by 637
  6. 2025Native Sparse Attention: Hardware-Aligned and Natively Trainable Sparse Attentioncited by 407
  7. 2023Learning to Compress Prompts with Gist Tokenscited by 390
  8. 2024MInference 1.0: Accelerating Pre-filling for Long-Context LLMs via Dynamic Sparse Attentioncited by 383
  9. 2024∞Bench: Extending Long Context Evaluation Beyond 100K Tokenscited by 367
  10. 2023Adapting Language Models to Compress Contextscited by 356
  11. 2024Magpie: Alignment Data Synthesis from Scratch by Prompting Aligned LLMs with Nothingcited by 346
  12. 2023Large Language Models for Information Retrieval: A Surveycited by 331
  13. 2023LongLoRA: Efficient Fine-tuning of Long-Context Large Language Modelscited by 244
  14. 2023Landmark Attention: Random-Access Infinite Context Length for Transformerscited by 213
  15. 2023In-context Autoencoder for Context Compression in a Large Language Modelcited by 199
  16. 2025MoBA: Mixture of Block Attention for Long-Context LLMscited by 198
  17. 2024InfLLM: Training-Free Long-Context Extrapolation for LLMs with an Efficient Context Memorycited by 197
  18. 2021BookSum: A Collection of Datasets for Long-form Narrative Summarizationcited by 184
  19. 2025MiniMax-01: Scaling Foundation Models with Lightning Attentioncited by 166
  20. 2024You Only Cache Once: Decoder-Decoder Architectures for Language Modelscited by 144
  21. 2025XAttention: Block Sparse Attention with Antidiagonal Scoringcited by 137
  22. 2024HELMET: How to Evaluate Long-Context Language Models Effectively and Thoroughlycited by 125
  23. 2024SeerAttention: Learning Intrinsic Sparse Attention in Your LLMscited by 124
  24. 2024Long Context Compression with Activation Beaconcited by 104
  25. 2025Tensor Product Attention Is All You Needcited by 31
  26. 2024A Silver Bullet or a Compromise for Full Attention? A Comprehensive Study of Gist Token-based Context Compressioncited by 17

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 9.

9 papers.

  1. 2026Learning to Evict from Key-Value Cachecited by 14
  2. 2026Read As Human: Compressing Context via Parallelizable Close Reading and Skimmingcited by 10
  3. 2026Dynamic Long Context Reasoning over Compressed Memory via End-to-End Reinforcement Learningcited by 8
  4. 2026Kwai Summary Attention Technical Reportcited by 7
  5. 2026Simplified Sparse Attention via Gist Tokenscited by 6
  6. 2026Fix the Structural Bottleneck: Context Compression via Explicit Information Transmissioncited by 2
  7. 2026MARCH: Scaling Recurrent Memory with Content-Routed State Anchorscited by 2
  8. 2026Safin-1: Safety from Within through Memory-Native State Evolutioncited by 1
  9. 2026Kwai Summary Attention V2 Technical Reportcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025RocketKV: Accelerating Long-Context LLM Inference via Two-Stage KV Cache Compression
  2. 2025Compactor: Calibrated Query-Agnostic KV Cache Compression with Approximate Leverage Scores
  3. 2025ParallelComp: Parallel Long-Context Compressor for Length Extrapolation
  4. 2025ChunkKV: Semantic-Preserving KV Cache Compression for Efficient Long-Context LLM Inference
  5. 2025TailorKV: A Hybrid Framework for Long-Context Inference via Tailored KV Cache Optimization
  6. 2026SeKV: Resolution-Adaptive KV Cache with Hierarchical Semantic Memory for Long-Context LLM Inference
  7. 2024Locret: Enhancing Eviction in Long-Context LLM Inference with Trained Retaining Heads on Consumer-Grade Devices
  8. 2026DepthWeave-KV: Token-Adaptive Cross-Layer Residual Factorization for Long-Context KV Cache Compression