mlsanityData to 14 Oct 2026

Prompt Compression via Activation Aggregation

arXiv 2607.08399 ↗First posted 9 Jul 2026cs.CL cs.LGcited by 0

On the map: Efficient LLM inference and serving → Prompt and context compression

Abstract (arXiv)

Large language models process prompts by propagating activations through dozens of layers before generating a response. We ask whether the task-relevant information contained in an instruction prompt can be compressed into a single activation vector and re-injected into the model, replacing the original token sequence? We show this is achievable using a learned weighted sum of activations extracted at an intermediate layer and injected at an early layer of the target LLM. The compressed vector preserves task-relevant information, incurring an accuracy drop of under 2% relative to full prompt processing. Beyond its practical implications, including reducing per-query computation for fixed instruction prompts without reprocessing the original token sequence, our analysis reveals structure in the activation space of LLMs: (i) mid-layer representations transfer meaningfully to early layers, suggesting a degree of cross-layer compatibility in how information is encoded; (ii) a single activation vector encodes a quantifiable and recoverable amount of semantic information; (iii) a weighted sum of activations is a robust representation compressor.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Prompt and context compression, one of 12 directions in Efficient LLM inference and serving. Builds on 16 of its 34 references that are in the map. First posted on arXiv 9 Jul 2026.

Builds on

16 of 34 references.

References · 16

  1. 2024The Llama 3 Herd of Modelscited by 14,715
  2. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,944
  3. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,886
  4. 2017Axiomatic Attribution for Deep Networkscited by 3,384
  5. 2023RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Controlcited by 3,080
  6. 2023Efficient Streaming Language Models with Attention Sinkscited by 2,186
  7. 2023Steering Llama 2 via Contrastive Activation Additioncited by 1,085
  8. 2023The Geometry of Truth: Emergent Linear Structure in Large Language Model Representations of True/False Datasetscited by 712
  9. 2022Transformer Feed-Forward Layers Build Predictions by Promoting Concepts in the Vocabulary Spacecited by 532
  10. 2023Learning to Compress Prompts with Gist Tokenscited by 390
  11. 2023Adapting Language Models to Compress Contextscited by 356
  12. 2023In-Context Learning Creates Task Vectorscited by 312
4 more
  1. 2024LayerSkip: Enabling Early Exit Inference and Self-Speculative Decodingcited by 307
  2. 2023In-context Vectors: Making In Context Learning More Effective and Controllable Through Latent Space Steeringcited by 245
  3. 2024Transcoders Find Interpretable LLM Feature Circuitscited by 200
  4. 2025How Well do LLMs Compress Their Own Chain-of-Thought? A Token Complexity Approachcited by 97

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025Understanding and Improving Information Preservation in Prompt Compression for LLMs
  2. 2025Task-agnostic Prompt Compression with Context-aware Sentence Embedding and Reward-guided Task Descriptor
  3. 2026Lexical Prompt Compression for Large Language Models: A Training-Free, Deterministic Pipeline with Empirical Pareto Analysis Across Eleven Task Categories
  4. 2024Style-Compress: An LLM-Based Prompt Compression Framework Considering Task-Specific Styles
  5. 2025Dynamic Compressing Prompts for Efficient Inference of Large Language Models
  6. 2024Prompt-SAW: Leveraging Relation-Aware Graphs for Textual Prompt Compression
  7. 2025Better Prompt Compression Without Multi-Layer Perceptrons
  8. 2025Learning a Single Token to Replace Long System Prompts in LLMs