mlsanityData to 7 Oct 2026

XC-Cache: Cross-Attending to Cached Context for Efficient LLM Inference

arXiv 2404.15420 ↗First posted 23 Apr 2024cs.CL cs.AIcited by 12

On the map: Efficient LLM inference and serving → KV cache optimization

Abstract (arXiv)

In-context learning (ICL) approaches typically leverage prompting to condition decoder-only language model generation on reference information. Just-in-time processing of a context is inefficient due to the quadratic cost of self-attention operations, and caching is desirable. However, caching transformer states can easily require almost as much space as the model parameters. When the right context isn't known in advance, caching ICL can be challenging. This work addresses these limitations by introducing models that, inspired by the encoder-decoder architecture, use cross-attention to condition generation on reference text without the prompt. More precisely, we leverage pre-trained decoder-only models and only train a small number of added layers. We use Question-Answering (QA) as a testbed to evaluate the ability of our models to perform conditional generation and observe that they outperform ICL, are comparable to fine-tuned prompted LLMs, and drastically reduce the space footprint relative to standard KV caching by two orders of magnitude.

Key facts. Cited by 12 papers in the map (data through 7 Oct 2026). Placed in KV cache optimization, one of 12 directions in Efficient LLM inference and serving. Builds on 40 of its 57 references that are in the map. First posted on arXiv 23 Apr 2024.

Builds on

40 of 57 references.

Closest ancestors · 1

Same direction, newest first

  1. 2024Keyformer: KV Cache Reduction through Key Tokens Selection for Efficient Generative Inference14 Mar

Other references · 39

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2020Language Models are Few-Shot Learnerscited by 32,199
  4. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  5. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  6. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  7. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,160
  8. 2019BERTScore: Evaluating Text Generation with BERTcited by 5,462
  9. 2018HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answeringcited by 3,686
  10. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,611
  11. 2023Lost in the Middle: How Language Models Use Long Contextscited by 3,383
  12. 2020Longformer: The Long-Document Transformercited by 3,362
27 more
  1. 2023Universal and Transferable Adversarial Attacks on Aligned Language Modelscited by 2,914
  2. 2023Efficient Streaming Language Models with Attention Sinkscited by 2,115
  3. 2019Language Models as Knowledge Bases?cited by 1,922
  4. 2022GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformerscited by 1,838
  5. 2016MS MARCO: A Human Generated MAchine Reading COmprehension Datasetcited by 1,591
  6. 2018Know What You Don't Know: Unanswerable Questions for SQuADcited by 1,538
  7. 2023GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpointscited by 1,311
  8. 2020Leveraging Passage Retrieval with Generative Models for Open Domain Question Answeringcited by 1,203
  9. 2023StarCoder: may the source be with you!cited by 790
  10. 2023In-Context Retrieval-Augmented Language Modelscited by 694
  11. 2019Latent Retrieval for Weakly Supervised Open Domain Question Answeringcited by 691
  12. 2023An Empirical Study of Catastrophic Forgetting in Large Language Models During Continual Fine-tuningcited by 622
  13. 2024StarCoder 2 and The Stack v2: The Next Generationcited by 484
  14. 2024LLM2Vec: Large Language Models Are Secretly Powerful Text Encoderscited by 396
  15. 2023Adapting Language Models to Compress Contextscited by 350
  16. 2023CodeT5+: Open Code Large Language Models for Code Understanding and Generationcited by 327
  17. 2022Efficient Training of Language Models to Fill in the Middlecited by 248
  18. 2024Generative Representational Instruction Tuningcited by 227
  19. 2023Evaluating Correctness and Faithfulness of Instruction-Following Models for Question Answeringcited by 150
  20. 2024Dynamic Memory Compression: Retrofitting LLMs for Accelerated Inferencecited by 115
  21. 2021TopiOCQA: Open-domain Conversational Question Answering with Topic Switchingcited by 95
  22. 2023Attention Sorting Combats Recency Bias In Long Context Language Modelscited by 89
  23. 2024Long-Context Language Modeling with Parallel Context Encodingcited by 84
  24. 2022On the Relation between Sensitivity and Accuracy in In-context Learningcited by 71
  25. 2023Shall We Pretrain Autoregressive Language Models with Retrieval? A Comprehensive Studycited by 63
  26. 2024In-Context Learning Can Re-learn Forbidden Taskscited by 5
  27. 2020ROME: Robustifying Memory-Efficient NAS via Topology Disentanglement and Gradient Accumulationcited by 3

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 4; 2025: 5; 2026: 3.

12 papers.

  1. 2024A Survey on Large Language Model Acceleration based on KV Cache Managementcited by 125
  2. 2024TurboRAG: Accelerating Retrieval-Augmented Generation with Precomputed KV Caches for Chunked Textcited by 60
  3. 2024RepLiQA: A Question-Answering Dataset for Benchmarking LLMs on Unseen Reference Contentcited by 26
  4. 2025Efficient Prompt Compression with Evaluator Heads for Long-Context Transformer Inferencecited by 10
  5. 2024Debug Smarter, Not Harder: AI Agents for Error Resolution in Computational Notebookscited by 5
  6. 2026Stochastic KV Routing: Enabling Adaptive Depth-Wise Cache Sharingcited by 3
  7. 2025Hierarchical Retrieval at Scale: Bridging Transparency and Efficiencycited by 1
  8. 2026CoMem: Reusing Transformer Depth across Queries with Persistent Intermediate Residualscited by 1
  9. 2025Explain-Query-Test: Self-Evaluating LLMs Via Explanation and Comprehension Discrepancycited by 0
  10. 2025Context-Aware Language Models for Forecasting Market Impact from Sequences of Financial Newscited by 0
  11. 2025HA-RAG: Hotness-Aware RAG Acceleration via Mixed Precision and Data Placementcited by 0
  12. 2026KV-Kaizen: Learning Context-Adaptive Cache Compression Choicescited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Training-Free Exponential Context Extension via Cascading KV Cache
  2. 2026Compressing Long Context into Answer-Aligned Memory Embeddings for LLM Inference
  3. 2025IC-Cache: Efficient Large Language Model Serving via In-context Caching
  4. 2025LLMs Know What to Drop: Self-Attention Guided KV Cache Eviction for Efficient Long-Context Inference
  5. 2024Quest: Query-Aware Sparsity for Efficient Long-Context LLM Inference
  6. 2026SeDeM: Selective Decompression of Hidden-State Memories for Long-Context Question Answering
  7. 2024UniICL: An Efficient Unified Framework Unifying Compression, Selection, and Generation
  8. 2024InfiniGen: Efficient Generative Inference of Large Language Models with Dynamic KV Cache Management