# Spectral-Window Hybrid (SWH)

arXiv: https://arxiv.org/abs/2601.01313 · HTML: https://mlsanity.com/paper/2601.01313/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 6 of its 6 references that are in the map.
- First posted on arXiv 4 Jan 2026.

## Builds on (most cited first, 6 of 6)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Mamba: Linear-Time Sequence Modeling with Selective State Spaces](https://mlsanity.com/paper/2312.00752/) (2023)
- [FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness](https://mlsanity.com/paper/2205.14135/) (2022)
- [Efficiently Modeling Long Sequences with Structured State Spaces](https://mlsanity.com/paper/2111.00396/) (2021)
- [Jamba: A Hybrid Transformer-Mamba Language Model](https://mlsanity.com/paper/2403.19887/) (2024)
- [Griffin: Mixing Gated Linear Recurrences with Local Attention for Efficient Language Models](https://mlsanity.com/paper/2402.19427/) (2024)

## Abstract (arXiv)

Scaling sequence modeling to extreme contexts requires balancing computational efficiency with representational expressivity. While Transformers provide precise retrieval via the attention mechanism, their quadratic O(T2) complexity limits their application to long-horizon tasks. In this work, we propose the Spectral-Window Hybrid (SWH), an architecture that decouples sequence modeling into two textitparallel streams: a global branch utilizing the Convolution Theorem to model long-range decay dynamics in O(T log T) time, and a local branch employing sliding-window attention for token interactions within a bounded context. By aggregating these representations, SWH avoids the computational bottleneck of global attention while retaining local precision. We demonstrate that SWH matches the perplexity of standard Transformers on short contexts while enabling efficient linear scaling to extended sequences. The code is available at https://github.com/VladimerKhasia/SWH