# SWAN-GPT: An Efficient and Scalable Approach for Long-Context Language Modeling

arXiv: https://arxiv.org/abs/2504.08719 · HTML: https://mlsanity.com/paper/2504.08719/

## Key facts

- Cited by 19 papers in the map (data through 14 Oct 2026).
- Builds on 35 of its 44 references that are in the map.
- First posted on arXiv 11 Apr 2025.

## Builds on (most cited first, 30 of 35)

- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [The Llama 3 Herd of Models](https://mlsanity.com/paper/2407.21783/) (2024)
- [Efficient Memory Management for Large Language Model Serving with PagedAttention](https://mlsanity.com/paper/2309.06180/) (2023)
- [Mamba: Linear-Time Sequence Modeling with Selective State Spaces](https://mlsanity.com/paper/2312.00752/) (2023)
- [RoFormer: Enhanced Transformer with Rotary Position Embedding](https://mlsanity.com/paper/2104.09864/) (2021)
- [Longformer: The Long-Document Transformer](https://mlsanity.com/paper/2004.05150/) (2020)
- [Efficient Streaming Language Models with Attention Sinks](https://mlsanity.com/paper/2309.17453/) (2023)
- [Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism](https://mlsanity.com/paper/1909.08053/) (2019)
- [Gemma 3 Technical Report](https://mlsanity.com/paper/2503.19786/) (2025)
- [Big Bird: Transformers for Longer Sequences](https://mlsanity.com/paper/2007.14062/) (2020)
- [RULER: What's the Real Context Size of Your Long-Context Language Models?](https://mlsanity.com/paper/2404.06654/) (2024)
- [Transformer Feed-Forward Layers Are Key-Value Memories](https://mlsanity.com/paper/2012.14913/) (2020)
- [DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Model](https://mlsanity.com/paper/2405.04434/) (2024)
- [RWKV: Reinventing RNNs for the Transformer Era](https://mlsanity.com/paper/2305.13048/) (2023)
- [Extending Context Window of Large Language Models via Positional Interpolation](https://mlsanity.com/paper/2306.15595/) (2023)
- [YaRN: Efficient Context Window Extension of Large Language Models](https://mlsanity.com/paper/2309.00071/) (2023)
- [KVQuant: Towards 10 Million Context Length LLM Inference with KV Cache Quantization](https://mlsanity.com/paper/2401.18079/) (2024)
- [The Impact of Positional Encoding on Length Generalization in Transformers](https://mlsanity.com/paper/2305.19466/) (2023)
- [Efficient Training of Language Models to Fill in the Middle](https://mlsanity.com/paper/2207.14255/) (2022)
- [Qwen2.5-1M Technical Report](https://mlsanity.com/paper/2501.15383/) (2025)
- [Nemotron-CC: Transforming Common Crawl into a Refined Long-Horizon Pretraining Dataset](https://mlsanity.com/paper/2412.02595/) (2024)
- [Transformer Language Models without Positional Encodings Still Learn Positional Information](https://mlsanity.com/paper/2203.16634/) (2022)
- [How to Train Long-Context Language Models (Effectively)](https://mlsanity.com/paper/2410.02660/) (2024)
- [LLM Maybe LongLM: Self-Extend LLM Context Window Without Tuning](https://mlsanity.com/paper/2401.01325/) (2024)
- [Scaling Laws of RoPE-based Extrapolation](https://mlsanity.com/paper/2310.05209/) (2023)
- [LM-Infinite: Zero-Shot Extreme Length Generalization for Large Language Models](https://mlsanity.com/paper/2308.16137/) (2023)
- [Long Context Compression with Activation Beacon](https://mlsanity.com/paper/2401.03462/) (2024)
- [Training-Free Long-Context Scaling of Large Language Models](https://mlsanity.com/paper/2402.17463/) (2024)
- [Length Generalization of Causal Transformers without Position Encoding](https://mlsanity.com/paper/2404.12224/) (2024)
- [Rope to Nope and Back Again: A New Hybrid Attention Strategy](https://mlsanity.com/paper/2501.18795/) (2025)

## Built on by (most cited first, 19 of 19)

- [Kimi Linear: An Expressive, Efficient Attention Architecture](https://mlsanity.com/paper/2510.26692/) (2025)
- [Four Over Six: More Accurate NVFP4 Quantization with Adaptive Block Scaling](https://mlsanity.com/paper/2512.02010/) (2025)
- [The Mean-Field Dynamics of Transformers](https://mlsanity.com/paper/2512.01868/) (2025)
- [Extending the Context of Pretrained LLMs by Dropping Their Positional Embeddings](https://mlsanity.com/paper/2512.12167/) (2025)
- [Critical attention scaling in long-context transformers](https://mlsanity.com/paper/2510.05554/) (2025)
- [Hybrid Linear Attention Done Right: Efficient Distillation and Effective Architectures for Extremely Long Contexts](https://mlsanity.com/paper/2601.22156/) (2026)
- [Optimizing Mixture of Block Attention](https://mlsanity.com/paper/2511.11571/) (2025)
- [SmallThinker: A Family of Efficient Large Language Models Natively Trained for Local Deployment](https://mlsanity.com/paper/2507.20984/) (2025)
- [Homogenized Transformers](https://mlsanity.com/paper/2604.01978/) (2026)
- [Rethinking the Role of Efficient Attention in Hybrid Architectures](https://mlsanity.com/paper/2606.15378/) (2026)
- [Periodic RoPE for Infinite Context LLMs](https://mlsanity.com/paper/2605.27980/) (2026)
- [Gaussian Equivalence for Self-Attention: Asymptotic Spectral Analysis of Attention Matrix](https://mlsanity.com/paper/2510.06685/) (2025)
- [Optimizing Native Sparse Attention with Latent Attention and Local Global Alternating Strategies](https://mlsanity.com/paper/2511.00819/) (2025)
- [An empirical study on the limitation of Transformers in program trace generation](https://mlsanity.com/paper/2509.25073/) (2025)
- [Frontier Language Models Struggle to Copy: Text Can Be Better Viewed in 2D](https://mlsanity.com/paper/2607.16072/) (2026)
- [How Local Mixing Encodes Relative Position in Global NoPE Attention](https://mlsanity.com/paper/2609.38109/) (2026)
- [Shifting Mechanisms: How Positional Encoding Choice Shapes In-Context Retrieval](https://mlsanity.com/paper/2609.38530/) (2026)
- [Mechanics of Long-Context Hybrid Models Part 1.1: From Hybrid Attention to Hybrid Position](https://mlsanity.com/paper/2610.10114/) (2026)
- [FreeSolo: Linear-time Sparse Attention with Zero-Shot Length Extrapolation](https://mlsanity.com/paper/2610.17191/) (2026)

## Abstract (arXiv)

We present a decoder-only Transformer architecture that robustly generalizes to sequence lengths substantially longer than those seen during training. Our model, SWAN-GPT, interleaves layers without positional encodings (NoPE) and sliding-window attention layers equipped with rotary positional encodings (SWA-RoPE). Experiments demonstrate strong performance on sequence lengths significantly longer than the training length without the need for additional long-context training. This robust length extrapolation is achieved through our novel architecture, enhanced by a straightforward dynamic scaling of attention scores during inference. In addition, SWAN-GPT is more computationally efficient than standard GPT architectures, resulting in cheaper training and higher throughput. Further, we demonstrate that existing pre-trained decoder-only models can be efficiently converted to the SWAN architecture with minimal continued training, enabling longer contexts. Overall, our work presents an effective approach for scaling language models to longer contexts in a robust and efficient manner.