mlsanityData to 14 Oct 2026

IAM: Efficient Inference through Attention Mapping between Different-scale LLMs

arXiv 2507.11953 ↗First posted 16 Jul 2025cs.CL cs.LGcited by 10

On the map: Efficient LLM inference and serving → KV cache optimization

Abstract (arXiv)

LLMs encounter significant challenges in resource consumption nowadays, especially with long contexts. Despite extensive efforts dedicate to enhancing inference efficiency, these methods primarily exploit internal sparsity within the models, without leveraging external information for optimization. We identify the high similarity of attention matrices across different-scale LLMs, which offers a novel perspective for optimization. We first conduct a comprehensive analysis of how to measure similarity, how to select mapping Layers and whether mapping is consistency. Based on these insights, we introduce the IAM framework, which achieves dual benefits of accelerated attention computation and reduced KV cache usage by performing attention mapping between small and large LLMs. Our experimental results demonstrate that IAM can accelerate prefill by 15% and reduce KV cache usage by 22.1% without appreciably sacrificing performance. Experiments on different series of models show the generalizability of IAM. Importantly, it is also orthogonal to many existing KV cache optimization methods, making it a versatile addition to the current toolkit for enhancing LLM efficiency.

Key facts. Cited by 10 papers in the map (data through 14 Oct 2026). Placed in KV cache optimization, one of 12 directions in Efficient LLM inference and serving. Builds on 20 of its 27 references that are in the map. First posted on arXiv 16 Jul 2025.

Builds on

20 of 27 references.

Closest ancestors · 4

Same direction, newest first

  1. 2024Model Tells You Where to Merge: Adaptive KV Cache Merging for LLMs on Long-Context Tasks11 Jul
  2. 2024PyramidInfer: Pyramid KV Cache Compression for High-throughput LLM Inference21 May
  3. 2023Efficient Memory Management for Large Language Model Serving with PagedAttention12 Sep
  4. 2023Scissorhands: Exploiting the Persistence of Importance Hypothesis for LLM KV Cache Compression at Test Time26 May

Other references · 16

  1. 2020Measuring Massive Multitask Language Understandingcited by 7,963
  2. 2018HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answeringcited by 3,754
  3. 2016Pointer Sentinel Mixture Modelscited by 3,096
  4. 2023FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioningcited by 2,598
  5. 2023Efficient Streaming Language Models with Attention Sinkscited by 2,186
  6. 2024Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phonecited by 1,917
  7. 2019Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelismcited by 1,868
  8. 2024Medusa: Simple LLM Inference Acceleration Framework with Multiple Decoding Headscited by 843
  9. 2024KVQuant: Towards 10 Million Context Length LLM Inference with KV Cache Quantizationcited by 591
  10. 2023FlexGen: High-Throughput Generative Inference of Large Language Models with a Single GPUcited by 575
  11. 2024Better & Faster Large Language Models via Multi-token Predictioncited by 360
  12. 2024LLMLingua-2: Data Distillation for Efficient and Faithful Task-Agnostic Prompt Compressioncited by 348
4 more
  1. 2021Efficient Attentions for Long Document Summarizationcited by 345
  2. 2024Mooncake: A KVCache-centric Disaggregated Architecture for LLM Servingcited by 196
  3. 2022Speculative Decoding: Exploiting Speculative Execution for Accelerating Seq2seq Generationcited by 171
  4. 2024You Only Cache Once: Decoder-Decoder Architectures for Language Modelscited by 144

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 3; 2026: 7.

10 papers.

  1. 2026Cross-Model KV Cache Transfer in LLM Families: A Closed-Form Linear Mapping for Prefill Reusecited by 15
  2. 2026CacheBridge: Efficient Cross-Model KV Cache Transfercited by 6
  3. 2025CoViPAL: Layer-wise Contextualized Visual Token Pruning for Large Vision-Language Modelscited by 3
  4. 2025A Layered Intuition -- Method Model with Scope Extension for LLM Reasoningcited by 3
  5. 2026From AR to Diffusion: Efficiently Adapting Large Language Models with Strictly Causal and Elastic Horizonscited by 3
  6. 2025From Parameters to Performance: A Data-Driven Study on LLM Structure and Developmentcited by 1
  7. 2026ProxyKV: Cross-Model Proxy Pruning for Efficient Long-Context LLM Inferencecited by 1
  8. 2026Emergent Structure in the Marginal Attention Space of Language Modelscited by 0
  9. 2026Accelerating Cross-Model LLM Inference via Attention Map Reusecited by 0
  10. 2026CacheLink: Functional KV Transport for Zero-Prefill Cross-Model Context Handoffcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026HybridGen: Efficient LLM Generative Inference via CPU-GPU Hybrid Computing
  2. 2024Beyond KV Caching: Shared Attention for Efficient LLMs
  3. 2025SmallKV: Small Model Assisted Compensation of KV Cache Compression for Efficient LLM Inference
  4. 2025HCAttention: Extreme KV Cache Compression via Heterogeneous Attention Computing for LLMs
  5. 2026Unifying Sparse Attention with Hierarchical Memory for Scalable Long-Context LLM Serving
  6. 2025Progressive Sparse Attention: Algorithm and System Co-design for Efficient Attention in LLM Serving
  7. 2026Sparse Attention as a Range Searching Problem: Towards an Inference-Efficient Index for KV Cache
  8. 2025MoSKA: Mixture of Shared KV Attention for Efficient Long-Sequence LLM Inference