mlsanityData to 7 Oct 2026

Long Context Compression with Activation Beacon

arXiv 2401.03462 ↗First posted 7 Jan 2024cs.CL cs.AIcited by 101

On the map: Efficient LLM inference and serving → Prompt and context compression

Abstract (arXiv)

Long context compression is a critical research problem due to its significance in reducing the high computational and memory costs associated with LLMs. In this paper, we propose Activation Beacon, a plug-in module for transformer-based LLMs that targets effective, efficient, and flexible compression of long contexts. To achieve this, our method introduces the following technical designs. 1) We directly compress the activations (i.e. keys and values at every layer), rather than leveraging soft prompts to relay information (which constitute a major bottleneck to encapsulate the complex information within long contexts). 2) We tailor the compression workflow, where each fine-grained input unit is progressively compressed, enabling high-quality compression and efficient computation during both training and inference. 3) We train the model through compression-based auto-regression, making full use of plain texts and instructional data to optimize the model's compression performance. 4) During training, we randomly sample a compression ratio at each step, teaching the model to support a wide range of compression configurations. Extensive evaluations are conducted on various long-context tasks whose lengths (e.g., 128K) may far exceed the maximum training length (20K), such as document understanding, few-shot learning, and Needle-in-a-Haystack. Whilst existing methods struggle to handle these challenging tasks, Activation Beacon maintains a comparable performance to the uncompressed baseline across various scenarios, achieving a 2x acceleration in inference time and an 8x reduction of memory costs for KV cache. Our data, model, and code have been released at urlhttps://github.com/FlagOpen/FlagEmbedding/.

Key facts. Cited by 101 papers in the map (data through 7 Oct 2026). Placed in Prompt and context compression, one of 12 directions in Efficient LLM inference and serving. Builds on 23 of its 38 references that are in the map. First posted on arXiv 7 Jan 2024.

Builds on

23 of 38 references.

Closest ancestors · 1

Same direction, newest first

  1. 2023LongLLMLingua: Accelerating and Enhancing LLMs in Long Context Scenarios via Prompt Compression10 Oct

Other references · 22

  1. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  2. 2020Measuring Massive Multitask Language Understandingcited by 7,768
  3. 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,581
  4. 2020Longformer: The Long-Document Transformercited by 3,362
  5. 2023FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioningcited by 2,551
  6. 2020The Pile: An 800GB Dataset of Diverse Text for Language Modelingcited by 2,346
  7. 2019HuggingFace's Transformers: State-of-the-art Natural Language Processingcited by 2,201
  8. 2019BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questionscited by 2,140
  9. 2023Efficient Streaming Language Models with Attention Sinkscited by 2,115
  10. 2020Big Bird: Transformers for Longer Sequencescited by 1,673
  11. 2023LongBench: A Bilingual, Multitask Benchmark for Long Context Understandingcited by 1,378
  12. 2023GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpointscited by 1,311
10 more
  1. 2023H2O: Heavy-Hitter Oracle for Efficient Generative Inference of Large Language Modelscited by 805
  2. 2023Extending Context Window of Large Language Models via Positional Interpolationcited by 721
  3. 2023YaRN: Efficient Context Window Extension of Large Language Modelscited by 584
  4. 2023Learning to Compress Prompts with Gist Tokenscited by 379
  5. 2023Adapting Language Models to Compress Contextscited by 350
  6. 2023LLMLingua: Compressing Prompts for Accelerated Inference of Large Language Modelscited by 279
  7. 2023LongLoRA: Efficient Fine-tuning of Long-Context Large Language Modelscited by 243
  8. 2023Scaling Transformer to 1M tokens and beyond with RMTcited by 97
  9. 2021Think you have Solved Direct-Answer Question Answering? Try ARC-DA, the Direct-Answer AI2 Reasoning Challengecited by 79
  10. 2023Compressed Context Memory For Online Language Model Interactioncited by 42

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 27; 2025: 25; 2026: 49.

101 papers.

Sort:

By area:

  1. 2024RULER: What's the Real Context Size of Your Long-Context Language Models?cited by 1,270
  2. 2024LongRoPE: Extending LLM Context Window Beyond 2 Million Tokenscited by 302
  3. 2024LongWriter: Unleashing 10,000+ Word Generation from Long Context LLMscited by 142
  4. 2025LightThinker: Thinking Step-by-Step Compressioncited by 104
  5. 2024TriForce: Lossless Acceleration of Long Sequence Generation with Hierarchical Speculative Decodingcited by 98
  6. 2024Eigen Attention: Attention in Low-Rank Space for KV Cache Compressioncited by 71
  7. 2024Training-Free Long-Context Scaling of Large Language Modelscited by 61
  8. 2024Length Generalization of Causal Transformers without Position Encodingcited by 46
  9. 2025Rope to Nope and Back Again: A New Hybrid Attention Strategycited by 46
  10. 2024LongAgent: Scaling Language Models to 128k Context through Multi-Agent Collaborationcited by 38
  11. 2024In Search of Needles in a 11M Haystack: Recurrent Memory Finds What LLMs Misscited by 36
  12. 2025Video-XL-2: Towards Very Long-Video Understanding Through Task-Aware KV Sparsificationcited by 36
38 more
  1. 2026Doc-to-LoRA: Learning to Instantly Internalize Contextscited by 36
  2. 2024V2PE: Improving Multimodal Long-Context Capability of Vision-Language Models with Variable Visual Position Encodingcited by 28
  3. 2024InfiniPot: Infinite Context Processing on Memory-Constrained LLMscited by 27
  4. 2024LongGenBench: Long-context Generation Benchmarkcited by 27
  5. 2025Eyes Wide Open: Ego Proactive Video-LLM for Streaming Videocited by 25
  6. 2026DeltaEvolve: Accelerating Scientific Discovery through Momentum-Driven Evolutioncited by 23
  7. 2025DAST: Context-Aware Compression in LLMs via Dynamic Allocation of Soft Tokenscited by 21
  8. 2025GMSA: Enhancing Context Compression via Group Merging and Layer Semantic Alignmentcited by 20
  9. 2024Recent advances in text embedding: A Comprehensive Review of Top-Performing Methods on the MTEB Benchmarkcited by 19
  10. 2024Lean Attention: Hardware-Aware Scalable Attention Mechanism for the Decode-Phase of Transformerscited by 16
  11. 2025SWAN-GPT: An Efficient and Scalable Approach for Long-Context Language Modelingcited by 16
  12. 2024BGE Landmark Embedding: A Chunking-Free Embedding Method For Retrieval Augmented Long-Context Large Language Modelscited by 15
  13. 2024Extending Llama-3's Context Ten-Fold Overnightcited by 15
  14. 2024A Silver Bullet or a Compromise for Full Attention? A Comprehensive Study of Gist Token-based Context Compressioncited by 15
  15. 2025Autoencoding-Free Context Compression for LLMs via Contextual Semantic Anchorscited by 14
  16. 20243D-RPE: Enhancing Long-Context Modeling Through 3D Rotary Position Encodingcited by 13
  17. 2026COMI: Coarse-to-fine Context Compression via Marginal Information Gaincited by 13
  18. 2024AI-native Memory: A Pathway from LLMs Towards AGIcited by 12
  19. 2026Learning to Evict from Key-Value Cachecited by 12
  20. 2025FAEDKV: Infinite-Window Fourier Transform for Unbiased KV Cache Compressioncited by 11
  21. 2026Toward Efficient Agents: Memory, Tool learning, and Planningcited by 11
  22. 2026Latent Context Compilation: Distilling Long Context into Compact Portable Memorycited by 11
  23. 2025SoLoPO: Unlocking Long-Context Capabilities in LLMs via Short-to-Long Preference Optimizationcited by 10
  24. 2026Read As Human: Compressing Context via Parallelizable Close Reading and Skimmingcited by 10
  25. 2024Position IDs Matter: An Enhanced Position Layout for Efficient Context Compression in Large Language Modelscited by 9
  26. 2025Sequential-NIAH: A Needle-In-A-Haystack Benchmark for Extracting Sequential Needles from Long Contextscited by 9
  27. 2025Memo: Training Memory-Efficient Embodied Agents with Reinforcement Learningcited by 9
  28. 2025UniGist: Towards General and Hardware-aligned Sequence-level Long Context Compressioncited by 8
  29. 2025FreqKV: Key-Value Compression in Frequency Domain for Context Window Extensioncited by 7
  30. 2025AdmTree: Compressing Lengthy Context with Adaptive Semantic Treescited by 7
  31. 2025Detecting Knowledge Boundary of Vision Large Language Models by Sampling-Based Inferencecited by 6
  32. 2025FlowKV: Enhancing Multi-Turn Conversational Coherence in LLMs via Isolated Key-Value Cache Managementcited by 6
  33. 2025No Mean Feat: Simple, Strong Baselines for Context Compressioncited by 6
  34. 2026Gated Differentiable Working Memory for Long-Context Language Modelingcited by 6
  35. 2026A Mechanistic View on Video Generation as World Models: State and Dynamicscited by 6
  36. 2026IntentKV: Cross-Turn Intent-Aware KV Cache Pruning for Agent Inferencecited by 6
  37. 2026CoMeT: Collaborative Memory Transformer for Efficient Long Context Modelingcited by 5
  38. 2026Dynamic Long Context Reasoning over Compressed Memory via End-to-End Reinforcement Learningcited by 5

50 most cited of 101 All 101 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024Recurrent Context Compression: Efficiently Expanding the Context Window of LLM
  2. 2023Context Compression for Auto-regressive Transformers with Sentinel Tokens
  3. 2026ATACompressor: Adaptive Task-Aware Compression for Efficient Long-Context Processing in LLMs
  4. 2025Efficient Prompt Compression with Evaluator Heads for Long-Context Transformer Inference
  5. 2024Enhancing and Accelerating Large Language Models via Instruction-Aware Contextual Compression
  6. 2026Prompt Compression in the Wild: Measuring Latency, Rate Adherence, and Quality for Faster LLM Inference
  7. 2025Compress, Gather, and Recompute: REFORMing Long-Context Processing in Transformers
  8. 2025Context Cascade Compression: Exploring the Upper Limits of Text Compression