mlsanityData to 14 Oct 2026

NLL-Guided Full-Attention Layer Selection for Training-Free Sliding-Window Adaptation

arXiv 2606.27791 ↗First posted 26 Jun 2026cs.CL cs.AIcited by 1

Not on the map. Why?

Abstract (arXiv)

Hybrid attention models that mix full and sliding-window attention across layers offer a promising approach to efficient long-context inference, but the critical question of which layers should retain full attention remains unsolved. Existing methods use either fixed periodic patterns or attention-based heuristics that may not capture what matters for downstream accuracy. We propose NLL-guided layer selection, a training-free method that directly measures each layer's importance by computing the negative log-likelihood degradation on answer tokens when that layer uses sliding-window instead of full attention. On LongMemEval with Qwen3-4B, our method achieves 64.6% accuracy using only 1/4 full-attention layers, matching the 1/2-FA periodic baseline (65.0%) while halving the computational budget. NLL-guided selection outperforms the SWAA-reported periodic 1/4-FA baseline by 10.4 percentage points and a matched LightTransfer-style baseline by 26.4 percentage points. De-confounding analysis shows the signal is consistent with long-range attention needs rather than generic layer sensitivity. The method requires only ~15 minutes of one-time calibration, advancing the efficiency-accuracy Pareto frontier for long-context LLM deployment.

Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Builds on 16 of its 17 references that are in the map. First posted on arXiv 26 Jun 2026.

Builds on

16 of 17 references.

References · 16

  1. 2025Qwen3 Technical Reportcited by 8,820
  2. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,944
  3. 2020Longformer: The Long-Document Transformercited by 3,413
  4. 2023FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioningcited by 2,598
  5. 2023Efficient Streaming Language Models with Attention Sinkscited by 2,186
  6. 2024Gemma 2: Improving Open Language Models at a Practical Sizecited by 1,998
  7. 2020Big Bird: Transformers for Longer Sequencescited by 1,684
  8. 2024SnapKV: LLM Knows What You are Looking for Before Generationcited by 880
  9. 2024LongMemEval: Benchmarking Chat Assistants on Long-Term Interactive Memorycited by 707
  10. 2024Quest: Query-Aware Sparsity for Efficient Long-Context LLM Inferencecited by 501
  11. 2024MInference 1.0: Accelerating Pre-filling for Long-Context LLMs via Dynamic Sparse Attentioncited by 383
  12. 2024Beyond the Limits: A Survey of Techniques to Extend the Context Length in Large Language Modelscited by 76
4 more
  1. 2025Distilling to Hybrid Attention Models via KL-Guided Layer Selectioncited by 20
  2. 2024LightTransfer: Your Long-Context LLM is Secretly a Hybrid Model with Effortless Adaptationcited by 12
  3. 2025SWAA: Sliding Window Attention Adaptation for Efficient and Quality Preserving Long Context Processingcited by 4
  4. 2025Training-free Context-adaptive Attention for Efficient Long Context Modelingcited by 1

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 1.

1 paper.

  1. 2026On-Demand Attention: Language Models Know When to Recallcited by 0