# MSWA: Refining Local Attention with Multi-ScaleWindow Attention

arXiv: https://arxiv.org/abs/2501.01039 · HTML: https://mlsanity.com/paper/2501.01039/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 2 Jan 2025.

## Built on by (most cited first, 5 of 5)

- [SoLA-Vision: Fine-grained Layer-wise Linear Softmax Hybrid Attention](https://mlsanity.com/paper/2601.11164/) (2026)
- [Customizing the Inductive Biases of Softmax Attention using Structured Matrices](https://mlsanity.com/paper/2509.07963/) (2025)
- [Depth-Staggered Fibonacci Spacing for Sparse Attention: Static Schedules Beat Learned Dilation and Extrapolate Where Dense Attention Fails](https://mlsanity.com/paper/2606.28560/) (2026)
- [MOSAIC: Adaptive Inter-layer Composition for Efficient Heterogeneous Vision-Language Models](https://mlsanity.com/paper/2607.09029/) (2026)
- [GLIDE: Guided Layerwise Hybrid Attention for Efficient LLM Inference](https://mlsanity.com/paper/2607.24788/) (2026)

## Abstract (arXiv)

Transformer-based LLMs have achieved exceptional performance across a wide range of NLP tasks. However, the standard self-attention mechanism suffers from quadratic time complexity and linearly increased cache size. Sliding window attention (SWA) solves this problem by restricting the attention range to a fixed-size local context window. Nevertheless, SWA employs a uniform window size for each head in each layer, making it inefficient in capturing context of varying scales. To mitigate this limitation, we propose Multi-Scale Window Attention (MSWA) which applies diverse window sizes across heads and layers in the Transformer. It not only allows for different window sizes among heads within the same layer but also progressively increases window size allocation from shallow to deep layers, thus enabling the model to capture contextual information with different lengths and distances. Experimental results on language modeling and common-sense reasoning tasks substantiate that MSWA outperforms traditional local attention in both effectiveness and efficiency.