# Cache the Encoder Within:Compact, Reusable Memory across LLM Queries

arXiv: https://arxiv.org/abs/2610.10058 · HTML: https://mlsanity.com/paper/2610.10058/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 12 of its 32 references that are in the map.
- First posted on arXiv 7 Oct 2026.

## Builds on (most cited first, 12 of 12)

- [Distilling the Knowledge in a Neural Network](https://mlsanity.com/paper/1503.02531/) (2015)
- [Qwen3 Technical Report](https://mlsanity.com/paper/2505.09388/) (2025)
- [RULER: What's the Real Context Size of Your Long-Context Language Models?](https://mlsanity.com/paper/2404.06654/) (2024)
- [Compressive Transformers for Long-Range Sequence Modelling](https://mlsanity.com/paper/1911.05507/) (2019)
- [CacheBlend: Fast Large Language Model Serving for RAG with Cached Knowledge Fusion](https://mlsanity.com/paper/2405.16444/) (2024)
- [CacheGen: KV Cache Compression and Streaming for Fast Large Language Model Serving](https://mlsanity.com/paper/2310.07240/) (2023)
- [Fast State Restoration in LLM Serving with HCache](https://mlsanity.com/paper/2410.05004/) (2024)
- [KV Packet: Recomputation-Free Context-Independent KV Caching for LLMs](https://mlsanity.com/paper/2604.13226/) (2026)
- [The Residual Stream Is All You Need: On the Redundancy of the KV Cache in Transformer Inference](https://mlsanity.com/paper/2603.19664/) (2026)
- [ILRe: Intermediate Layer Retrieval for Context Compression in Causal Language Models](https://mlsanity.com/paper/2508.17892/) (2025)
- [SemPIC: Learning Semantic Position-Independent KV Caches](https://mlsanity.com/paper/2607.28069/) (2026)
- [CoMem: Reusing Transformer Depth across Queries with Persistent Intermediate Residuals](https://mlsanity.com/paper/2607.28263/) (2026)

## Abstract (arXiv)

Repeated queries over shared documents incur redundant encoding, while caching model states introduces persistent storage costs. Building on CoMem's intermediate-state interface, EncBank treats a pretrained LLM's lower layers as a reusable document encoder and compactly stores their outputs for an adapted upper-layer reader. A self-distilled suffix adapter is shared across storage precisions within each backbone, without quantization-specific retraining. Across five benchmark suites on three Qwen backbones spanning different sizes and full-attention and hybrid architectures, 4-bit storage keeps each reported benchmark aggregate within one score point of native-precision EncBank. In a fixed Qwen3-8B workload, it retains 28.1% of the native-precision persistent GPU store. Separate native-precision controls yield a 1.40x selected-pack prefill speedup over same-evidence, same-adapter text replay, at a 3.12-point RULER accuracy cost. A native-precision Qwen3.8-27B configuration also passes 70 of 89 Terminal-Bench 2.1 tasks. EncBank thus combines reusable computation with compact memory, while task fidelity and end-to-end benefits remain dependent on the workload, preparation costs, and reuse frequency.