mlsanityData to 14 Oct 2026

Compress, Gather, and Recompute: REFORMing Long-Context Processing in Transformers

arXiv 2506.01215 ↗First posted 1 Jun 2025cs.CL cs.LGcited by 3

On the map: Efficient LLM inference and serving → Early exit and edge inference

Abstract (arXiv)

As large language models increasingly gain popularity in real-world applications, processing extremely long contexts, often exceeding the model's pre-trained context limits, has emerged as a critical challenge. While existing approaches to efficient long-context processing show promise, recurrent compression-based methods struggle with information preservation, whereas random access approaches require substantial memory resources. We introduce REFORM, a novel inference framework that efficiently handles long contexts through a two-phase approach. First, it incrementally processes input chunks while maintaining a compressed KV cache, constructs cross-layer context embeddings, and utilizes early exit strategy for improved efficiency. Second, it identifies and gathers essential tokens via similarity matching and selectively recomputes the KV cache. Compared to baselines, REFORM achieves over 52% and 34% performance gains on RULER and BABILong respectively at 1M context length. It also outperforms baselines on Infinite-Bench, RepoEval, and MM-NIAH, demonstrating flexibility across diverse tasks and domains. Additionally, REFORM reduces inference time by 30% and peak memory usage by 5%, achieving both efficiency and superior performance.

Key facts. Cited by 3 papers in the map (data through 14 Oct 2026). Placed in Early exit and edge inference, one of 12 directions in Efficient LLM inference and serving. Builds on 47 of its 59 references that are in the map. First posted on arXiv 1 Jun 2025.

Builds on

47 of 59 references.

Closest ancestors · 1

Same direction, newest first

  1. 2024Transformers are Multi-State RNNs11 Jan

Other references · 46

  1. 2024The Llama 3 Herd of Modelscited by 14,715
  2. 2024Qwen2.5 Technical Reportcited by 4,942
  3. 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,703
  4. 2018HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answeringcited by 3,754
  5. 2023Mistral 7Bcited by 3,164
  6. 2016Pointer Sentinel Mixture Modelscited by 3,096
  7. 2023FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioningcited by 2,598
  8. 2023Efficient Streaming Language Models with Attention Sinkscited by 2,186
  9. 2019Transformer-XL: Attentive Language Models Beyond a Fixed-Length Contextcited by 1,850
  10. 2024RULER: What's the Real Context Size of Your Long-Context Language Models?cited by 1,346
  11. 2024SnapKV: LLM Knows What You are Looking for Before Generationcited by 880
  12. 2019Analyzing Multi-Head Self-Attention: Specialized Heads Do the Heavy Lifting, the Rest Can Be Prunedcited by 856
34 more
  1. 2023H2O: Heavy-Hitter Oracle for Efficient Generative Inference of Large Language Modelscited by 832
  2. 2023Extending Context Window of Large Language Models via Positional Interpolationcited by 740
  3. 2022MTEB: Massive Text Embedding Benchmarkcited by 640
  4. 2023YaRN: Efficient Context Window Extension of Large Language Modelscited by 601
  5. 2024KVQuant: Towards 10 Million Context Length LLM Inference with KV Cache Quantizationcited by 591
  6. 2023RepoCoder: Repository-Level Code Completion Through Iterative Retrieval and Generationcited by 367
  7. 2024∞Bench: Extending Long Context Evaluation Beyond 100K Tokenscited by 367
  8. 2025Layer by Layer: Uncovering Hidden Representations in Language Modelscited by 311
  9. 2023Improving Text Embeddings with Large Language Modelscited by 303
  10. 2024DuoAttention: Efficient Long-Context LLM Inference with Retrieval and Streaming Headscited by 288
  11. 2023LongLoRA: Efficient Fine-tuning of Long-Context Large Language Modelscited by 244
  12. 2024Retrieval Head Mechanistically Explains Long-Context Factualitycited by 232
  13. 2024BABILong: Testing the Limits of LLMs with Long Context Reasoning-in-a-Haystackcited by 215
  14. 2023Landmark Attention: Random-Access Infinite Context Length for Transformerscited by 213
  15. 2022Recurrent Memory Transformercited by 203
  16. 2024Leave No Context Behind: Efficient Infinite Context Transformers with Infini-attentioncited by 200
  17. 2023Augmenting Language Models with Long-Term Memorycited by 189
  18. 2024Pixtral 12Bcited by 179
  19. 2024GEAR: An Efficient KV Cache Compression Recipe for Near-Lossless Generative Inference of LLMcited by 158
  20. 2024Retrieval Augmented Generation or Long-Context LLMs? A Comprehensive Study and Hybrid Approachcited by 119
  21. 2023PoSE: Efficient Context Window Extension of LLMs via Positional Skip-wise Trainingcited by 111
  22. 2024RazorAttention: Efficient KV Cache Compression Through Retrieval Headscited by 100
  23. 2024Model Tells You Where to Merge: Adaptive KV Cache Merging for LLMs on Long-Context Taskscited by 96
  24. 2024Loki: Low-rank Keys for Efficient Sparse Attentioncited by 89
  25. 2024Get More with LESS: Synthesizing Recurrence with KV Cache Compression for Efficient LLM Inferencecited by 75
  26. 2024Attention Heads of Large Language Models: A Surveycited by 67
  27. 2024Needle In A Multimodal Haystackcited by 58
  28. 2024In Defense of RAG in the Era of Long-Context Language Modelscited by 53
  29. 2022An Analysis of Fusion Functions for Hybrid Retrievalcited by 51
  30. 2024Identifying Semantic Induction Heads to Understand In-Context Learningcited by 47
  31. 2024Hierarchical Context Merging: Better Long Context Understanding for Pre-trained LLMscited by 30
  32. 2024InfiniPot: Infinite Context Processing on Memory-Constrained LLMscited by 27
  33. 2022Out-of-Domain Semantics to the Rescue! Zero-Shot Hybrid Retrieval Modelscited by 26
  34. 2024ReAttention: Training-Free Infinite Context with Finite Attention Scopecited by 13

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 1; 2026: 2.

3 papers.

  1. 2025A Survey of Context Engineering for Large Language Modelscited by 126
  2. 2026CoMem: Reusing Transformer Depth across Queries with Persistent Intermediate Residualscited by 1
  3. 2026RelaxKV: Recomputation Guided by the Query with Sparse Context Attention for Efficient KV Cache Reusecited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Recurrent Context Compression: Efficiently Expanding the Context Window of LLM
  2. 2023Context Compression for Auto-regressive Transformers with Sentinel Tokens
  3. 2025LLMCache: Layer-Wise Caching Strategies for Accelerated Reuse in Transformer Inference
  4. 2024KV Cache Compression, But What Must We Give in Return? A Comprehensive Benchmark of Long Context Capable Approaches
  5. 2025KV Pareto: Systems-Level Optimization of KV Cache and Model Compression for Long Context Inference
  6. 2024Enhancing and Accelerating Large Language Models via Instruction-Aware Contextual Compression
  7. 2026Training Transformers for KV Cache Compressibility
  8. 2026End-to-End Context Compression at Scale