mlsanityData to 14 Oct 2026

Efficient Many-Shot In-Context Learning with Dynamic Block-Sparse Attention

arXiv 2503.08640 ↗First posted 11 Mar 2025cs.CLcited by 12

Not on the map. Why?

Abstract (arXiv)

Many-shot in-context learning has recently shown promise as an alternative to finetuning, with the major advantage that the same model can be served for multiple tasks. However, this shifts the computational burden from training-time to inference-time, making deployment of many-shot ICL challenging to justify in-practice. This cost is further increased if a custom demonstration set is retrieved for each inference example. We present Dynamic Block-Sparse Attention, a training-free framework for retrieval-based many-shot in-context learning. By combining carefully designed block-sparse attention and retrieval of cached groups of demonstrations, we achieve comparable per-example latency to finetuning while maintaining on average >95% of the best method's accuracy across strong ICL and finetuning baselines. We hope that this will further enable the deployment of many-shot ICL at scale.

Key facts. Cited by 12 papers in the map (data through 14 Oct 2026). Builds on 32 of its 44 references that are in the map. First posted on arXiv 11 Mar 2025.

Builds on

32 of 44 references.

References · 32

  1. 2020Measuring Massive Multitask Language Understandingcited by 7,963
  2. 2023Lost in the Middle: How Language Models Use Long Contextscited by 3,455
  3. 2023FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioningcited by 2,598
  4. 2023Efficient Streaming Language Models with Attention Sinkscited by 2,186
  5. 2020Big Bird: Transformers for Longer Sequencescited by 1,684
  6. 2019Generating Long Sequences with Sparse Transformerscited by 1,546
  7. 2023GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpointscited by 1,346
  8. 2024SnapKV: LLM Knows What You are Looking for Before Generationcited by 880
  9. 2016Efficient and robust approximate nearest neighbor search using Hierarchical Navigable Small World graphscited by 861
  10. 2022A Survey on In-context Learningcited by 721
  11. 2019SAMSum Corpus: A Human-annotated Dialogue Dataset for Abstractive Summarizationcited by 535
  12. 2020Efficient Intent Detection with Dual Sentence Encoderscited by 433
20 more
  1. 2019An Evaluation Dataset for Intent Classification and Out-of-Scope Predictioncited by 386
  2. 2024DuoAttention: Efficient Long-Context LLM Inference with Retrieval and Streaming Headscited by 288
  3. 2018Online Learning: A Comprehensive Surveycited by 205
  4. 2024Flex Attention: A Programming Model for Generating Optimized Attention Kernelscited by 185
  5. 2023ZeroSCROLLS: A Zero-Shot Benchmark for Long Text Understandingcited by 176
  6. 2019Benchmarking Natural Language Understanding Services for building Conversational Agentscited by 166
  7. 2023S-LoRA: Serving Thousands of Concurrent LoRA Adapterscited by 152
  8. 2023Few-shot Fine-tuning vs. In-context Learning: A Fair Comparison and Evaluationcited by 136
  9. 2024In-Context Learning with Long-Context Models: An In-Depth Explorationcited by 128
  10. 2024HELMET: How to Evaluate Long-Context Language Models Effectively and Thoroughlycited by 125
  11. 2024Long-Context LLMs Meet RAG: Overcoming Challenges for Long Inputs in RAGcited by 123
  12. 2022Parallel Context Windows for Large Language Modelscited by 81
  13. 2024TurboRAG: Accelerating Retrieval-Augmented Generation with Precomputed KV Caches for Chunked Textcited by 63
  14. 2025ChunkKV: Semantic-Preserving KV Cache Compression for Efficient Long-Context LLM Inferencecited by 60
  15. 2022Prototypical Calibration for Few-shot Learning of Language Modelscited by 58
  16. 2023In-Context Learning with Many Demonstration Examplescited by 39
  17. 2024Star Attention: Efficient LLM Inference over Long Sequencescited by 25
  18. 2024Compress then Serve: Serving Thousands of LoRA Adapters with Little Overheadcited by 24
  19. 2024Deeper Insights Without Updates: The Power of In-Context Learning Over Fine-Tuningcited by 21
  20. 2024When Precision Meets Position: BFloat16 Breaks Down RoPE in Long-Context Trainingcited by 15

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 2; 2026: 10.

12 papers.

  1. 2025Prompt-MII: Meta-Learning Instruction Induction for LLMscited by 9
  2. 2025You Only Fine-tune Once: Many-Shot In-Context Fine-Tuning for Large Language Modelscited by 3
  3. 2026Addressing Corpus Knowledge Poisoning Attacks on RAG Using Sparse Attentioncited by 3
  4. 2026KARA: Efficient Reasoning LLM Serving via Sliding-Window KV Cache Compressioncited by 2
  5. 2026AdapShot: Adaptive Many-Shot In-Context Learning with Semantic-Aware KV Cache Reusecited by 1
  6. 2026DepthWeave-KV: Token-Adaptive Cross-Layer Residual Factorization for Long-Context KV Cache Compressioncited by 1
  7. 2026GraphLoom: Reliability-Calibrated Graph Evidence Routing for Multimodal KG-RAGcited by 1
  8. 2026FreqDepthKV: Frequency-Guided Depth Sharing for Robust KV Cache Compression in Long-Context LLM Inferencecited by 0
  9. 2026Long-Context Demonstration Selection Using State Space Modelscited by 0
  10. 2026No More Free Lunch: Corpus Task Complexity Matters as Corpora Growcited by 0
  11. 2026MILO: Efficient Many-shot In-Context Learning with Block-wise Low-rank Compressioncited by 0
  12. 2026HHR: Hierarchical Hash Retrieval for Efficient LLM Generationcited by 0