mlsanityData to 14 Oct 2026

Accelerating Cross-Model LLM Inference via Attention Map Reuse

arXiv 2610.14688 ↗First posted 10 Oct 2026cs.LGcited by 0

On the map: Efficient LLM inference and serving → KV cache optimization also fits: Serving systems and request scheduling

Abstract (arXiv)

LLM-augmented data workflows often involve multiple heterogeneous models operating over substantial shared context, such as schemas, tool specifications, interaction history, and intermediate results. Although prefix caching can reduce prefilling overhead, existing approaches are largely limited to reuse within the same model because KV states are tightly coupled to model-specific architectures. We present MapFlow, a system for efficient heterogeneous LLM serving via cross-model attention map reuse. Instead of reusing KV states, MapFlow reuses attention maps, which capture token dependency patterns and are more transferable across models. To enable efficient and accurate reuse, we combine partial recomputation with lightweight linear translation across layers and heads. To make attention reuse practical for long contexts, we further introduce block-level sparse compression and a fused retrieval kernel. Experiments show that MapFlow reduces prefilling latency and improves end-to-end serving efficiency while preserving generation quality across heterogeneous model pairs and multi-turn routing workloads. These results suggest that attention maps provide an effective reusable state for heterogeneous LLM serving, especially in prefix-heavy data workflows. Our code is available: https://github.com/PKU-DAIR/MapFlow.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in KV cache optimization, one of 12 directions in Efficient LLM inference and serving. Builds on 50 of its 83 references that are in the map. First posted on arXiv 10 Oct 2026.

Builds on

50 of 83 references.

Closest ancestors · 9

Same direction, newest first

  1. 2026Multi-Segment Attention: Enabling Efficient KV-Cache Management for Faster Large Language Model Serving1 Jun
  2. 2026IceCache: Memory-efficient KV-cache Management for Long-Sequence LLMs12 Apr
  3. 2026ParisKV: Fast and Drift-Robust KV-Cache Retrieval for Long-Context LLMs7 Feb
  4. 2025Cache-to-Cache: Direct Semantic Communication Between Large Language Models3 Oct
  5. 2025IAM: Efficient Inference through Attention Mapping between Different-scale LLMs16 Jul
  6. 2024Quest: Query-Aware Sparsity for Efficient Long-Context LLM Inference16 Jun
  7. 2024CacheBlend: Fast Large Language Model Serving for RAG with Cached Knowledge Fusion26 May
  8. 2023CacheGen: KV Cache Compression and Streaming for Fast Large Language Model Serving11 Oct
  9. 2023Efficient Memory Management for Large Language Model Serving with PagedAttention12 Sep

Other references · 41

  1. 2017Attention Is All You Needcited by 60,361
  2. 2023Efficient Streaming Language Models with Attention Sinkscited by 2,186
  3. 2023LongBench: A Bilingual, Multitask Benchmark for Long Context Understandingcited by 1,417
  4. 2024RULER: What's the Real Context Size of Your Long-Context Language Models?cited by 1,346
  5. 2023SGLang: Efficient Execution of Structured Language Model Programscited by 1,141
  6. 2023ChatDev: Communicative Agents for Software Developmentcited by 919
  7. 2023FrugalGPT: How to Use Large Language Models While Reducing Cost and Improving Performancecited by 853
  8. 2024RouteLLM: Learning to Route LLMs with Preference Datacited by 682
  9. 2023LLM-Blender: Ensembling Large Language Models with Pairwise Ranking and Generative Fusioncited by 564
  10. 2024MInference 1.0: Accelerating Pre-filling for Long-Context LLMs via Dynamic Sparse Attentioncited by 383
  11. 2023SpecInfer: Accelerating Generative Large Language Model Serving with Tree-based Speculative Inference and Verificationcited by 365
  12. 2024Personal LLM Agents: Insights and Survey about the Capability, Efficiency and Securitycited by 260
29 more
  1. 2024RouterBench: A Benchmark for Multi-LLM Routing Systemcited by 238
  2. 2023SARATHI: Efficient LLM Inference by Piggybacking Decodes with Chunked Prefillscited by 193
  3. 2024Llumnix: Dynamic Scheduling for Large Language Model Servingcited by 174
  4. 2023S-LoRA: Serving Thousands of Concurrent LoRA Adapterscited by 152
  5. 2023ChatDB: Augmenting LLMs with Databases as Their Symbolic Memorycited by 140
  6. 2021Scatterbrain: Unifying Sparse and Low-rank Attention Approximationcited by 130
  7. 2025FlexPrefill: A Context-Aware Sparse Attention Mechanism for Efficient Long-Sequence Inferencecited by 123
  8. 2026Ministral 3cited by 114
  9. 2023Towards Efficient Generative Large Language Model Serving: A Survey from Algorithms to Systemscited by 102
  10. 2024PQCache: Product Quantization-based KVCache for Long Context LLM Inferencecited by 95
  11. 2023SpotServe: Serving Generative Large Language Models on Preemptible Instancescited by 87
  12. 2024A Unified Approach to Routing and Cascading for LLMscited by 80
  13. 2024Helix: Serving Large Language Models over Heterogeneous GPUs and Network via Max-Flowcited by 55
  14. 2024MuxServe: Flexible Spatial-Temporal Multiplexing for Multiple LLM Servingcited by 52
  15. 2024λ-Tune: Harnessing Large Language Models for Automated Database System Tuningcited by 19
  16. 2026Latent Space Communication via K-V Cache Alignmentcited by 17
  17. 2025Beyond Quacking: Deep Integration of Language Models and RAG into DuckDBcited by 15
  18. 2022LlamaTune: Sample-Efficient DBMS Configuration Tuningcited by 14
  19. 2025Can Large Language Models Be Query Optimizer for Relational Databases?cited by 11
  20. 2026Efficient LLM Serving for Agentic Workflows: A Data Systems Perspectivecited by 11
  21. 2022PromptEM: Prompt-tuning for Low-resource Generalized Entity Matchingcited by 10
  22. 2024A Survey of Large Language Model-Based Generative AI for Text-to-SQL: Benchmarks, Applications, Use Cases, and Challengescited by 10
  23. 2025Q-KVComm: Efficient Multi-Agent Communication Via Adaptive KV Cache Compressioncited by 9
  24. 2024Large Language Model Enhanced Knowledge Representation Learning: A Surveycited by 7
  25. 2025A Survey on Collaborating Small and Large Language Models for Performance, Cost-effectiveness, Cloud-edge Privacy, and Trustworthinesscited by 7
  26. 2025PICE: A Semantic-Driven Progressive Inference System for LLM Serving in Cloud-Edge Networkscited by 6
  27. 2025AlayaDB: The Data Foundation for Efficient and Effective Long-context LLM Inferencecited by 3
  28. 2026Coral: Cost-Efficient Multi-LLM Serving over Heterogeneous Cloud GPUscited by 2
  29. 2024LLMs as On-demand Customizable Servicecited by 1

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025MoSKA: Mixture of Shared KV Attention for Efficient Long-Sequence LLM Inference
  2. 2024Beyond KV Caching: Shared Attention for Efficient LLMs
  3. 2025Kascade: A Practical Sparse Attention Method for Long-Context LLM Inference
  4. 2026PackInfer: Compute- and I/O-Efficient Attention for Batched LLM Inference
  5. 2026C2KV: Compressed and Composable KV Cache Reuse for Efficient LLM Inference
  6. 2025Long-Context Modeling with Dynamic Hierarchical Sparse Attention for Memory-Constrained LLM Inference
  7. 2026You Only Index Once: Cross-Layer Sparse Attention with Shared Routing
  8. 2025Efficient Multi-Adapter LLM Serving via Cross-Model KV-Cache Reuse with Activated LoRA