mlsanityData to 7 Oct 2026

BOSCH: Black-Box Binary Optimization for Short-Context Attention-Head Selection in LLMs

arXiv 2604.05942 ↗First posted 7 Apr 2026cs.CLcited by 0

On the map: Efficient LLM inference and serving → KV cache optimization

Abstract (arXiv)

Post-training hybridization of large language models (LLMs) often replaces quadratic self-attention with sliding-window attention (SWA) to reduce KV cache usage and improve latency. Existing hybridization schemes are typically defined either at the layer level (e.g., interleaving) or at the head level via static rankings from local to global. Layer-level schemes ignore that local and global dependencies are routed through heads within the same layer, while static head-level rankings suffer from entanglement: a head's local/global behavior can change after hybridization. We propose BOSCH, Black-box Binary Optimization for Short-context Head Selection, a training-free method that formulates the problem as a Large Neighborhood Search and decomposes it into three subproblems: (i) layer-importance detection via small-budget black-box probes, (ii) adaptive per-layer SWA-ratio assignment based on these sensitivities, and (iii) grouped head-level optimization within ratio buckets. Extensive experiments on 4 LLMs ranging from 1.7B to 30B parameters, across 4 SWA ratios, show that BOSCH consistently outperforms layer-level heuristics and 6 strong static head-level methods, with larger gains at higher SWA ratios. Under continual pretraining, BOSCH recover original long-context performance faster and to a higher level. Analysis of the selected heads reveals substantial turnover for BOSCH across different SWA ratios, underscoring the importance of performing head-level selection for each target ratio rather than relying on fixed locality rankings.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in KV cache optimization, one of 12 directions in Efficient LLM inference and serving. Builds on 32 of its 53 references that are in the map. First posted on arXiv 7 Apr 2026.

Builds on

32 of 53 references.

Closest ancestors · 1

Same direction, newest first

  1. 2024RazorAttention: Efficient KV Cache Compression Through Retrieval Heads22 Jul

Other references · 31

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  4. 2020Measuring Massive Multitask Language Understandingcited by 7,768
  5. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,710
  6. 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,581
  7. 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layercited by 3,413
  8. 2020Longformer: The Long-Document Transformercited by 3,362
  9. 2019PIQA: Reasoning about Physical Commonsense in Natural Languagecited by 2,818
  10. 2023FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioningcited by 2,551
  11. 2023LongBench: A Bilingual, Multitask Benchmark for Long Context Understandingcited by 1,378
  12. 2023GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpointscited by 1,311
19 more
  1. 2024Transformers are SSMs: Generalized Models and Efficient Algorithms Through Structured State Space Dualitycited by 1,283
  2. 2024RULER: What's the Real Context Size of Your Long-Context Language Models?cited by 1,270
  3. 2025gpt-oss-120b & gpt-oss-20b Model Cardcited by 1,258
  4. 2017Mixed Precision Trainingcited by 1,135
  5. 2019What Does BERT Look At? An Analysis of BERT's Attentioncited by 994
  6. 2022In-context Learning and Induction Headscited by 930
  7. 2018A Tutorial on Bayesian Optimizationcited by 734
  8. 2024FlashAttention-3: Fast and Accurate Attention with Asynchrony and Low-precisioncited by 507
  9. 2024Gated Delta Networks: Improving Mamba2 with Delta Rulecited by 483
  10. 2024Retrieval Head Mechanistically Explains Long-Context Factualitycited by 219
  11. 2022A Fast Post-Training Pruning Framework for Transformerscited by 161
  12. 2025MiniMax-01: Scaling Foundation Models with Lightning Attentioncited by 157
  13. 2024How to Train Long-Context Language Models (Effectively)cited by 141
  14. 2024Liger Kernel: Efficient Triton Kernels for LLM Trainingcited by 123
  15. 2024The Mamba in the Llama: Distilling and Accelerating Hybrid Modelscited by 120
  16. 2025Zebra-Llama: Towards Extremely Efficient Hybrid Modelscited by 18
  17. 2020Telling BERT's full story: from Local Attention to Global Aggregationcited by 15
  18. 2025Unveiling Simplicities of Attention: Adaptive Long-Context Head Identificationcited by 5
  19. 2021JABER and SABER: Junior and Senior Arabic BERtcited by 3

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026GLIDE: Guided Layerwise Hybrid Attention for Efficient LLM Inference
  2. 2024Cross-layer Attention Sharing for Pre-trained Large Language Models
  3. 2024DHA: Learning Decoupled-Head Attention from Transformer Checkpoints via Adaptive Heads Fusion
  4. 2026Rethinking the Role of Efficient Attention in Hybrid Architectures
  5. 2024Beyond Linear Approximations: A Novel Pruning Approach for Attention Matrix
  6. 2026LycheeDecode: Accelerating Long-Context LLM Inference via Hybrid-Head Sparse Decoding
  7. 2026Flux Attention: Context-Aware Hybrid Attention for Efficient LLMs Inference
  8. 2026ConSA: Controllable Sparsity in Hybrid Attention via Learnable Allocation