Sliding Window Recurrences for Sequence Models
Not on the map. Why?
Abstract (arXiv)
Multi-hybrid architectures are poised to take over language modeling due to better quality and performance. We introduce a hierarchical decomposition framework for linear recurrences that allows us to develop algorithms aligned with GPU memory hierarchies, yielding Sliding Window Recurrences. We focus specifically on truncating recurrences to hardware-aligned windows which are naturally jagged, limiting costly inter-warp communication. Using SWR, we develop Phalanx layers that serve as drop-in replacements for windowed attention or linear recurrences. In 1B parameter multi-hybrid models, Phalanx achieves over 10-40% speedup across 4K to 32K context length over optimized Transformers while matching perplexity.
Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Builds on 24 of its 39 references that are in the map. First posted on arXiv 15 Dec 2025.
Builds on
24 of 39 references.
References · 24
- 2020Language Models are Few-Shot Learnerscited by 32,199
- 2024The Llama 3 Herd of Modelscited by 14,192
- 2025Qwen3 Technical Reportcited by 8,195
- 2023Mamba: Linear-Time Sequence Modeling with Selective State Spacescited by 4,616
- 2020Longformer: The Long-Document Transformercited by 3,362
- 2023FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioningcited by 2,551
- 2020Big Bird: Transformers for Longer Sequencescited by 1,673
- 2023GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpointscited by 1,311
- 2024Transformers are SSMs: Generalized Models and Efficient Algorithms Through Structured State Space Dualitycited by 1,283
- 2025gpt-oss-120b & gpt-oss-20b Model Cardcited by 1,258
- 2022Simplified State Space Layers for Sequence Modelingcited by 776
- 2023Gated Linear Attention Transformers with Hardware-Efficient Trainingcited by 490
12 more
- 2024Gated Delta Networks: Improving Mamba2 with Delta Rulecited by 483
- 2023Hyena Hierarchy: Towards Larger Convolutional Language Modelscited by 388
- 2024Simple linear attention language models balance the recall-throughput tradeoffcited by 197
- 2017Parallelizing Linear Recurrent Neural Nets Over Sequence Lengthcited by 126
- 2021CKConv: Continuous Kernel Convolution For Sequential Datacited by 117
- 2024Mechanistic Design and Scaling of Hybrid Architecturescited by 57
- 2025A Systematic Analysis of Hybrid Linear Attentioncited by 38
- 2023Laughing Hyena Distillery: Extracting Compact Recurrences From Convolutionscited by 24
- 2025Systems and Algorithms for Convolutional Multi-Hybrid Language Models at Scalecited by 12
- 2025Exploring Diffusion Transformer Designs via Graftingcited by 12
- 2025RATTENTION: Towards the Minimal Sliding Window Size in Local-Global Attention Modelscited by 10
- 2024STAR: Synthesis of Tailored Architecturescited by 7
Built on by
Citing papers per year: 2026: 1.
1 paper.