mlsanityData to 7 Oct 2026

xLSTM: Extended Long Short-Term Memory

arXiv 2405.04517 ↗First posted 7 May 2024cs.LG cs.AI stat.MLcited by 320

Not on the map. Why?

Cited by 38 in State-space sequence models (S4, Mamba), but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

In the 1990s, the constant error carousel and gating were introduced as the central ideas of the Long Short-Term Memory (LSTM). Since then, LSTMs have stood the test of time and contributed to numerous deep learning success stories, in particular they constituted the first Large Language Models (LLMs). However, the advent of the Transformer technology with parallelizable self-attention at its core marked the dawn of a new era, outpacing LSTMs at scale. We now raise a simple question: How far do we get in language modeling when scaling LSTMs to billions of parameters, leveraging the latest techniques from modern LLMs, but mitigating known limitations of LSTMs? Firstly, we introduce exponential gating with appropriate normalization and stabilization techniques. Secondly, we modify the LSTM memory structure, obtaining: (i) sLSTM with a scalar memory, a scalar update, and new memory mixing, (ii) mLSTM that is fully parallelizable with a matrix memory and a covariance update rule. Integrating these LSTM extensions into residual block backbones yields xLSTM blocks that are then residually stacked into xLSTM architectures. Exponential gating and modified memory structures boost xLSTM capabilities to perform favorably when compared to state-of-the-art Transformers and State Space Models, both in performance and scaling.

Key facts. Cited by 320 papers in the map (data through 7 Oct 2026). First posted on arXiv 7 May 2024.

Builds on

0 of 0 references.

No references here yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 77; 2025: 147; 2026: 96.

320 papers.

Sort:

By area:

  1. 2024Transformers are SSMs: Generalized Models and Efficient Algorithms Through Structured State Space Dualitycited by 1,283
  2. 2024FlashAttention-3: Fast and Accurate Attention with Asynchrony and Low-precisioncited by 507
  3. 2024Gated Delta Networks: Improving Mamba2 with Delta Rulecited by 483
  4. 2024Parallelizing Linear Transformers with the Delta Rule over Sequence Lengthcited by 373
  5. 2024Learning to (Learn at Test Time): RNNs with Expressive Hidden Statescited by 329
  6. 2024Titans: Learning to Memorize at Test Timecited by 303
  7. 2024You Only Cache Once: Decoder-Decoder Architectures for Language Modelscited by 135
  8. 2025TTT3R: 3D Reconstruction as Test-Time Trainingcited by 122
  9. 2025FlexPrefill: A Context-Aware Sparse Attention Mechanism for Efficient Long-Sequence Inferencecited by 121
  10. 2024The Mamba in the Llama: Distilling and Accelerating Hybrid Modelscited by 120
  11. 2025TiRex: Zero-Shot Forecasting Across Long and Short Horizons with Enhanced In-Context Learningcited by 107
  12. 2024MeshXL: Neural Coordinate Field for Generative 3D Foundation Modelscited by 100
38 more
  1. 2024Unlocking State-Tracking in Linear RNNs Through Negative Eigenvaluescited by 97
  2. 2024A Review of Large Language Models and Autonomous Agents in Chemistrycited by 96
  3. 2025Cartridges: Lightweight and general-purpose long context representations via self-studycited by 74
  4. 2025DeltaProduct: Improving State-Tracking in Linear RNNs via Householder Productscited by 69
  5. 2025Nested Learning: The Illusion of Deep Learning Architecturescited by 69
  6. 2024Transformers to SSMs: Distilling Quadratic Knowledge to Subquadratic Modelscited by 66
  7. 2024Inference Scaling for Long-Context Retrieval Augmented Generationcited by 65
  8. 2025Test-time regression: a unifying framework for designing sequence models with associative memorycited by 63
  9. 2024Longhorn: State Space Models are Amortized Online Learnerscited by 58
  10. 2025Forgetting Transformer: Softmax Attention with a Forget Gatecited by 50
  11. 2024DeciMamba: Exploring the Length Extrapolation Potential of Mambacited by 45
  12. 2025MoM: Linear Sequence Modeling with Mixture-of-Memoriescited by 44
  13. 2024LinFusion: 1 GPU, 1 Minute, 16K Imagecited by 42
  14. 2025On Vanishing Gradients, Over-Smoothing, and Over-Squashing in GNNs: Bridging Recurrent and Graph Learningcited by 42
  15. 2025Baichuan-M1: Pushing the Medical Capability of Large Language Modelscited by 41
  16. 2025Log-Linear Attentioncited by 39
  17. 2024LION: Linear Group RNN for 3D Object Detection in Point Cloudscited by 38
  18. 2024Towards Scalable and Stable Parallelization of Nonlinear RNNscited by 38
  19. 2025MesaNet: Sequence Modeling by Locally Optimal Test-Time Trainingcited by 37
  20. 2024Were RNNs All We Needed?cited by 36
  21. 2025LLMs are Greedy Agents: Effects of RL Fine-tuning on Decision-Making Abilitiescited by 36
  22. 2024DAPE: Data-Adaptive Positional Encoding for Length Extrapolationcited by 30
  23. 2024Marconi: Prefix Caching for the Era of Hybrid LLMscited by 29
  24. 2025SeerAttention-R: Sparse Attention Adaptation for Long Reasoningcited by 28
  25. 2024Understanding the differences in Foundation Models: Attention, State Space Models, and Recurrent Neural Networkscited by 26
  26. 2024B'MOJO: Hybrid State Space Realizations of Foundation Models with Eidetic and Fading Memorycited by 26
  27. 2024LoLCATs: On Low-Rank Linearizing of Large Language Modelscited by 25
  28. 2024Quamba: A Post-Training Quantization Recipe for Selective State Space Modelscited by 25
  29. 2025Thinking Slow, Fast: Scaling Inference Compute with Distilled Reasonerscited by 22
  30. 2025Primus: Enforcing Attention Usage for 3D Medical Image Segmentationcited by 22
  31. 2024Rethinking Conventional Wisdom in Machine Learning: From Generalization to Scalingcited by 21
  32. 2025Implicit Language Models are RNNs: Balancing Parallelization and Expressivitycited by 21
  33. 2025Mantis: Lightweight Foundation Model for Time Series Classificationcited by 21
  34. 2025Efficient Attention Mechanisms for Large Language Models: A Surveycited by 21
  35. 2026Toto 2.0: Time Series Forecasting Enters the Scaling Eracited by 21
  36. 2024Spectra: Surprising Effectiveness of Pretraining Ternary Language Models at Scalecited by 20
  37. 2024Retrieval-Augmented Decision Transformer: External Memory for In-context RLcited by 20
  38. 2025Understanding and Improving Length Generalization in Recurrent Modelscited by 19

50 most cited of 320 All 320 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025Overflow Prevention Enhances Long-Context Recurrent LLMs
  2. 2026Out of the Memory Barrier: A Highly Memory Efficient Training System for LLMs with Million-Token Contexts
  3. 2025LServe: Efficient Long-sequence LLM Serving with Unified Sparse Attention
  4. 2023LLM in a flash: Efficient Large Language Model Inference with Limited Memory
  5. 2024InfLLM: Training-Free Long-Context Extrapolation for LLMs with an Efficient Context Memory
  6. 2025Why Do Some Inputs Break Low-Bit LLM Quantization?
  7. 2025Pimba: A Processing-in-Memory Acceleration for Post-Transformer Large Language Model Serving
  8. 2023LongLLMLingua: Accelerating and Enhancing LLMs in Long Context Scenarios via Prompt Compression