mlsanityData to 14 Oct 2026

FreeSolo: Linear-time Sparse Attention with Zero-Shot Length Extrapolation

arXiv 2610.17191 ↗First posted 12 Oct 2026cs.LG cs.AIcited by 0

Not on the map. Why?

Abstract (arXiv)

Linear-time sequence models typically store long-context information in a recurrent state. A fixed-size state limits how much information can be stored, while hybrid architectures that add full attention for better recall incur quadratic sequence cost. Alternatively, sparse attention with continued pretraining (CPT) can extend the effective context length. Generalization beyond the trained length remains challenging, however, and CPT can be costly. We introduce FreeSolo, a native sparse attention mechanism with a short-range sliding window branch and a long-range branch that selects keypoint cells for direct token attention. Its linear-time variant, FS-L, learns to prune the prefix before query-dependent selection through short CPT. FS-B searches balanced trees with O(Nlog2 N) algorithmic cost including radix-sort construction. FreeSolo demonstrates better zero-shot length extrapolation than the baselines without long-context fine-tuning.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 43 of its 50 references that are in the map. First posted on arXiv 12 Oct 2026.

Builds on

43 of 50 references.

References · 43

  1. 2017Attention Is All You Needcited by 60,361
  2. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  3. 2023Mamba: Linear-Time Sequence Modeling with Selective State Spacescited by 4,718
  4. 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,703
  5. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,673
  6. 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layercited by 3,509
  7. 2023FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioningcited by 2,598
  8. 2023Efficient Streaming Language Models with Attention Sinkscited by 2,186
  9. 2020Transformers are RNNs: Fast Autoregressive Transformers with Linear Attentioncited by 2,022
  10. 2020GLU Variants Improve Transformercited by 1,804
  11. 2019Root Mean Square Layer Normalizationcited by 1,480
  12. 2024RULER: What's the Real Context Size of Your Long-Context Language Models?cited by 1,346
31 more
  1. 2024Transformers are SSMs: Generalized Models and Efficient Algorithms Through Structured State Space Dualitycited by 1,336
  2. 2024The FineWeb Datasets: Decanting the Web for the Finest Text Data at Scalecited by 1,160
  3. 2021Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolationcited by 1,103
  4. 2023Extending Context Window of Large Language Models via Positional Interpolationcited by 740
  5. 2023YaRN: Efficient Context Window Extension of Large Language Modelscited by 601
  6. 2024Gated Delta Networks: Improving Mamba2 with Delta Rulecited by 532
  7. 2024Quest: Query-Aware Sparsity for Efficient Long-Context LLM Inferencecited by 501
  8. 2025Native Sparse Attention: Hardware-Aligned and Natively Trainable Sparse Attentioncited by 407
  9. 2023The Impact of Positional Encoding on Length Generalization in Transformerscited by 327
  10. 2022Memorizing Transformerscited by 226
  11. 2023Landmark Attention: Random-Access Infinite Context Length for Transformerscited by 213
  12. 2025MoBA: Mixture of Block Attention for Long-Context LLMscited by 198
  13. 2024InfLLM: Training-Free Long-Context Extrapolation for LLMs with an Efficient Context Memorycited by 197
  14. 2025Kimi Linear: An Expressive, Efficient Attention Architecturecited by 196
  15. 2024Data Engineering for Scaling Language Models to 128K Contextcited by 186
  16. 2022Overcoming a Theoretical Limitation of Self-Attentioncited by 113
  17. 2024A Simple and Effective L2 Norm-Based Strategy for KV Cache Compressioncited by 86
  18. 2024Gated Slot Attention for Efficient Linear-Time Sequence Modelingcited by 76
  19. 2024What is Wrong with Perplexity for Long-context Language Modeling?cited by 69
  20. 2025Rope to Nope and Back Again: A New Hybrid Attention Strategycited by 47
  21. 2025MoM: Linear Sequence Modeling with Mixture-of-Memoriescited by 47
  22. 2022Dissecting Transformer Length Extrapolation via the Lens of Receptive Field Analysiscited by 42
  23. 2021Towards mental time travel: a hierarchical memory for reinforcement learning agentscited by 40
  24. 2024HMT: Hierarchical Memory Transformer for Efficient Long Context Language Processingcited by 21
  25. 2025Long-Context Generalization with Sparse Attentioncited by 21
  26. 2025SWAN-GPT: An Efficient and Scalable Approach for Long-Context Language Modelingcited by 19
  27. 2022Treeformer: Dense Gradient Trees for Efficient Attention Computationcited by 9
  28. 2024Efficient Length-Generalizable Attention via Causal Retrieval for Long-Context Language Modelingcited by 9
  29. 2025Hardware-aligned Hierarchical Sparse Attention for Efficient Long-term Memory Accesscited by 9
  30. 2026Raven: High-Recall Sequence Modeling with Sparse Memory Routingcited by 6
  31. 2026Neural Attention Search Linear: Towards Adaptive Token-Level Hybrid Attention Modelscited by 3

Built on by

Not cited yet.