# Better Prompt Compression Without Multi-Layer Perceptrons

arXiv: https://arxiv.org/abs/2501.06730 · HTML: https://mlsanity.com/paper/2501.06730/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Placed in Prompt and context compression, one of 12 directions in Efficient LLM inference and serving.
- Builds on 15 of its 23 references that are in the map.
- First posted on arXiv 12 Jan 2025.

## Placed in

- Efficient LLM inference and serving / [Prompt and context compression](https://mlsanity.com/inference/prompt-and-context-compression/)

## Builds on (most cited first, 15 of 15)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [LoRA: Low-Rank Adaptation of Large Language Models](https://mlsanity.com/paper/2106.09685/) (2021)
- [Layer Normalization](https://mlsanity.com/paper/1607.06450/) (2016)
- [LongLLMLingua: Accelerating and Enhancing LLMs in Long Context Scenarios via Prompt Compression](https://mlsanity.com/paper/2310.06839/) (2023)
- [Learning to Compress Prompts with Gist Tokens](https://mlsanity.com/paper/2304.08467/) (2023)
- [Adapting Language Models to Compress Contexts](https://mlsanity.com/paper/2305.14788/) (2023)
- [LLMLingua: Compressing Prompts for Accelerated Inference of Large Language Models](https://mlsanity.com/paper/2310.05736/) (2023)
- [Many-Shot In-Context Learning](https://mlsanity.com/paper/2404.11018/) (2024)
- [In-context Autoencoder for Context Compression in a Large Language Model](https://mlsanity.com/paper/2307.06945/) (2023)
- [Compressing Context to Enhance Inference Efficiency of Large Language Models](https://mlsanity.com/paper/2310.06201/) (2023)
- [Pretrained Transformers as Universal Computation Engines](https://mlsanity.com/paper/2103.05247/) (2021)
- [In-Context Learning with Long-Context Models: An In-Depth Exploration](https://mlsanity.com/paper/2405.00200/) (2024)
- [Prompt Compression and Contrastive Conditioning for Controllability and Toxicity Reduction in Language Models](https://mlsanity.com/paper/2210.03162/) (2022)
- [500xCompressor: Generalized Prompt Compression for Large Language Models](https://mlsanity.com/paper/2408.03094/) (2024)

## Built on by (most cited first, 4 of 4)

- [Mapping Text to Multiplex Graph: Prompt Compression as L'evy Walk-Guided Graph Pruning](https://mlsanity.com/paper/2607.01241/) (2026)
- [AquiLLM: An Architecture for Supporting Tacit Knowledge Capture in Research Groups](https://mlsanity.com/paper/2608.08883/) (2026)
- [Efficient Modular Learning through Naive LoRA Summation: Leveraging Orthogonality in High-Dimensional Models](https://mlsanity.com/paper/2508.11985/) (2025)
- [Learning from Research: Toward Lifelong Agent Harness Evolution](https://mlsanity.com/paper/2609.40169/) (2026)

## Abstract (arXiv)

Prompt compression is a promising approach to speeding up language model inference without altering the generative model. Prior works compress prompts into smaller sequences of learned tokens using an encoder that is trained as a LowRank Adaptation (LoRA) of the inference language model. However, we show that the encoder does not need to keep the original language model's architecture to achieve useful compression. We introduce the Attention-Only Compressor (AOC), which learns a prompt compression encoder after removing the multilayer perceptron (MLP) layers in the Transformer blocks of a language model, resulting in an encoder with roughly 67% less parameters compared to the original model. Intriguingly we find that, across a range of compression ratios up to 480x, AOC can better regenerate prompts and outperform a baseline compression encoder that is a LoRA of the inference language model without removing MLP layers. These results demonstrate that the architecture of prompt compression encoders does not need to be identical to that of the original decoder language model, paving the way for further research into architectures and approaches for prompt compression.