CoMem: Reusing Transformer Depth across Queries with Persistent Intermediate Residuals
Not on the map. Why?
Abstract (arXiv)
Repeated queries over shared documents repeatedly execute the same lower transformer layers. We introduce CoMem, which makes split depth j an explicit reusable-context axis: write one depth-j residual per token, select a bounded chunk set, and resume only layers [j:L). Among document-reuse systems we are aware of, CoMem jointly makes split depth a tunable serving axis and isolates it with a matched j=0 endpoint. On Qwen3-8B, j=12 reduces selected-pack Read from 931.9 to 664.4 ms (1.403x), with a 3.12-point RULER cost (95% CI [2.36, 3.93]); a continuous-prefix oracle recovers the full gap. The resulting depth axis quantifies a quality-latency-storage trade-off; a separate same-adapter, Write-inclusive pipeline is 2.74x faster. Equal-latency raw replay leads by 11.56 points with BM25, directly measuring an applicability boundary of prepaid depth rather than hiding it. CoMem stores 8 KiB/token versus 144 KiB/token for a protocol-aligned same-Qwen3 CacheBlend-style diagnostic; the cohorts and adaptation budgets are not matched. A context-position factorization identifies missing lower-layer document context as the dominant tested multikey error, and a 32-token overlap raises 92.5 to 98.5 without increasing persistent bytes or per-query Read. CoMem opens transformer depth as a measurable, tunable dimension for repeated-query long-context serving.
Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Builds on 41 of its 47 references that are in the map. First posted on arXiv 30 Jul 2026.
Builds on
41 of 47 references.
References · 41
- 2024The Llama 3 Herd of Modelscited by 14,192
- 2015Distilling the Knowledge in a Neural Networkcited by 10,076
- 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,160
- 2025Qwen3 Technical Reportcited by 8,195
- 2023Efficient Streaming Language Models with Attention Sinkscited by 2,115
- 2023LongBench: A Bilingual, Multitask Benchmark for Long Context Understandingcited by 1,378
- 2024RULER: What's the Real Context Size of Your Long-Context Language Models?cited by 1,270
- 2024SnapKV: LLM Knows What You are Looking for Before Generationcited by 848
- 2023H2O: Heavy-Hitter Oracle for Efficient Generative Inference of Large Language Modelscited by 805
- 2024Evaluating Very Long-Term Conversational Memory of LLM Agentscited by 800
- 2019Compressive Transformers for Long-Range Sequence Modellingcited by 699
- 2024PyramidKV: Dynamic KV Cache Compression based on Pyramidal Information Funnelingcited by 420
29 more
- 2023Prompt Cache: Modular Attention Reuse for Low-Latency Inferencecited by 255
- 2024CacheBlend: Fast Large Language Model Serving for RAG with Cached Knowledge Fusioncited by 211
- 2024BABILong: Testing the Limits of LLMs with Long Context Reasoning-in-a-Haystackcited by 211
- 2024InfLLM: Training-Free Long-Context Extrapolation for LLMs with an Efficient Context Memorycited by 192
- 2023Augmenting Language Models with Long-Term Memorycited by 186
- 2024MiniCache: KV Cache Compression in Depth Dimension for Large Language Modelscited by 122
- 2023Retrieval meets Long Context Large Language Modelscited by 114
- 2024RAGCache: Efficient Knowledge Caching for Retrieval-Augmented Generationcited by 97
- 2024MEMORYLLM: Towards Self-Updatable Large Language Modelscited by 89
- 2025Cartridges: Lightweight and general-purpose long context representations via self-studycited by 74
- 2024TurboRAG: Accelerating Retrieval-Augmented Generation with Precomputed KV Caches for Chunked Textcited by 60
- 2024EPIC: Efficient Position-Independent Caching for Serving Large Language Modelscited by 45
- 2024Discovering the Gems in Early Layers: Accelerating Long-Context LLMs with 1000x Input Token Reductioncited by 39
- 2025Cache-Craft: Managing Chunk-Caches for Efficient Retrieval-Augmented Generationcited by 36
- 2024Block-Attention for Efficient Prefillingcited by 34
- 2024Fast State Restoration in LLM Serving with HCachecited by 32
- 2025APE: Faster and Longer Context-Augmented Generation via Adaptive Parallel Encodingcited by 29
- 2024LLoCO: Learning Long Contexts Offlinecited by 16
- 2024XC-Cache: Cross-Attending to Cached Context for Efficient LLM Inferencecited by 12
- 2025MEPIC: Memory Efficient Position Independent Caching for LLM Servingcited by 10
- 2026KV Packet: Recomputation-Free Context-Independent KV Caching for LLMscited by 10
- 2026Cartridges at Scale: Training Modular KV Caches over Large Document Collectionscited by 6
- 2022Embedding Recycling for Language Modelscited by 5
- 2026From Prefix Cache to Fusion RAG Cache: Accelerating LLM Inference in Retrieval-Augmented Generationcited by 5
- 2020ReadOnce Transformers: Reusable Representations of Text for Transformerscited by 4
- 2026The Residual Stream Is All You Need: On the Redundancy of the KV Cache in Transformer Inferencecited by 4
- 2025Compress, Gather, and Recompute: REFORMing Long-Context Processing in Transformerscited by 3
- 2025ILRe: Intermediate Layer Retrieval for Context Compression in Causal Language Modelscited by 3
- 2025LLMCache: Layer-Wise Caching Strategies for Accelerated Reuse in Transformer Inferencecited by 1
Built on by
Citing papers per year: 2026: 1.
1 paper.