On Subquadratic Architectures: From Applications to Principles
Not on the map. Why?
Abstract (arXiv)
Transformers dominate modern sequence modeling, but their quadratic attention incurs substantial computational cost. Subquadratic architectures offer a scalable alternative. However, it remains unclear which designs yield the most effective sequence models. We compare three leading approaches: xLSTM, Mamba-2, and Gated DeltaNet. We evaluate these models on tasks with complex dependencies: (1) code-model pre-training, (2) distillation of code models from large language models, and (3) pre-training of time-series foundation models. Across these settings, xLSTM delivers the strongest overall performance. To explain xLSTM's advantage, we present a unified formulation and analyze the underlying architectural mechanisms, focusing on state tracking and memory dynamics. Our results show that xLSTM enables more flexible and stable memory correction via its gating scheme. We corroborate these findings on controlled synthetic length-generalization tasks. Overall, our findings indicate that xLSTM's gains on complex tasks stem from robust state tracking and accumulation.
Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Builds on 52 of its 63 references that are in the map. First posted on arXiv 10 Jun 2026.
Builds on
52 of 63 references.
References · 52
- 2017Attention Is All You Needcited by 59,932
- 2015Distilling the Knowledge in a Neural Networkcited by 10,076
- 2016Layer Normalizationcited by 5,579
- 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,203
- 2023Mamba: Linear-Time Sequence Modeling with Selective State Spacescited by 4,616
- 2014Neural Machine Translation by Jointly Learning to Align and Translatecited by 4,530
- 2014Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translationcited by 4,016
- 2023Let's Verify Step by Stepcited by 3,598
- 2020Longformer: The Long-Document Transformercited by 3,362
- 2021Efficiently Modeling Long Sequences with Structured State Spacescited by 2,424
- 2023Efficient Streaming Language Models with Attention Sinkscited by 2,115
- 2020Transformers are RNNs: Fast Autoregressive Transformers with Linear Attentioncited by 1,979
38 more
- 2020GLU Variants Improve Transformercited by 1,724
- 2018Group Normalizationcited by 1,514
- 2024Transformers are SSMs: Generalized Models and Efficient Algorithms Through Structured State Space Dualitycited by 1,283
- 2024Chronos: Learning the Language of Time Seriescited by 712
- 2022GPT-NeoX-20B: An Open-Source Autoregressive Language Modelcited by 711
- 2023A decoder-only foundation model for time-series forecastingcited by 650
- 2024Unified Training of Universal Time Series Forecasting Transformerscited by 588
- 2023Gated Linear Attention Transformers with Hardware-Efficient Trainingcited by 490
- 2021Linear Transformers Are Secretly Fast Weight Programmerscited by 471
- 2024Parallelizing Linear Transformers with the Delta Rule over Sequence Lengthcited by 373
- 2022Diagonal State Spaces are as Effective as Structured State Spacescited by 350
- 2024xLSTM: Extended Long Short-Term Memorycited by 320
- 2022Neural Networks and the Chomsky Hierarchycited by 214
- 2022Transformer Quality in Linear Timecited by 189
- 2024GIFT-Eval: A Benchmark For General Time Series Forecasting Model Evaluationcited by 188
- 2024The Illusion of State in State-Space Modelscited by 181
- 2025Kimi Linear: An Expressive, Efficient Attention Architecturecited by 169
- 2024Samba: Simple Hybrid State Space Models for Efficient Unlimited Context Language Modelingcited by 167
- 2024The Mamba in the Llama: Distilling and Accelerating Hybrid Modelscited by 120
- 2024Zamba: A Compact 7B SSM Hybrid Modelcited by 118
- 2018On the Practical Computational Power of Finite Precision RNNs for Language Recognitioncited by 117
- 2025TiRex: Zero-Shot Forecasting Across Long and Short Horizons with Enhanced In-Context Learningcited by 107
- 2024Unlocking State-Tracking in Linear RNNs Through Negative Eigenvaluescited by 97
- 2025Nemotron 3 Nano: Open, Efficient Mixture-of-Experts Hybrid Mamba-Transformer Model for Agentic Reasoningcited by 85
- 2023Exposing Attention Glitches with Flip-Flop Language Modelingcited by 74
- 2025DeltaProduct: Improving State-Tracking in Linear RNNs via Householder Productscited by 69
- 2024Transformers to SSMs: Distilling Quadratic Knowledge to Subquadratic Modelscited by 66
- 2019Sequential Neural Networks as Automatacited by 54
- 2024Linearizing Large Language Modelscited by 51
- 2024Toto: Time Series Optimized Transformer for Observabilitycited by 44
- 2025Llamba: Scaling Distilled Recurrent Models for Efficient Language Processingcited by 35
- 2025Liger: Linearizing Large Language Models to Gated Recurrent Structurescited by 32
- 2025RADLADS: Rapid Attention Distillation to Linear Attention Decoders at Scalecited by 24
- 2025Nemotron-Math: Efficient Long-Context Distillation of Mathematical Reasoning from Multi-Mode Supervisioncited by 22
- 2025xLSTM 7B: A Recurrent LLM for Fast and Efficient Inferencecited by 17
- 2026Why Are Linear RNNs More Parallelizable?cited by 7
- 2026Learning State-Tracking from Code Using Linear RNNscited by 5
- 2025xLSTM Scaling Laws: Competitive Performance with Linear Time-Complexitycited by 3
50 most cited of 52 All 52 (JSON)
Built on by
Citing papers per year: 2026: 1.
1 paper.
Similar, not cited
Close in wording, no citation either way
- 2025Numerical Investigation of Sequence Modeling Theory using Controllable Memory Functions
- 2025Pimba: A Processing-in-Memory Acceleration for Post-Transformer Large Language Model Serving
- 2026Long-Context Aware Upcycling: A New Frontier for Hybrid LLM Scaling
- 2024Enhanced Structured State Space Models via Grouped FIR Filtering and Attention Sink Mechanisms
- 2025Design Principles for Sequence Models via Coefficient Dynamics
- 2025Overflow Prevention Enhances Long-Context Recurrent LLMs
- 2026Why Depth Matters in Parallelizable Sequence Models: A Lie Algebraic View
- 2025Log-Linear Attention