# LISM: Long-range Integrative State space Models via Input-Latent State Interactions

arXiv: https://arxiv.org/abs/2509.04226 · HTML: https://mlsanity.com/paper/2509.04226/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Placed in Core SSM architectures for language, one of 10 directions in State-space sequence models (S4, Mamba).
- Builds on 31 of its 50 references that are in the map.
- First posted on arXiv 4 Sep 2025.

## Placed in

- State-space sequence models (S4, Mamba) / [Core SSM architectures for language](https://mlsanity.com/ssm/core-ssm-architectures-for-language/)

## Builds on (most cited first, 30 of 31)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks](https://mlsanity.com/paper/2005.11401/) (2020)
- [Mamba: Linear-Time Sequence Modeling with Selective State Spaces](https://mlsanity.com/paper/2312.00752/) (2023)
- [Pointer Sentinel Mixture Models](https://mlsanity.com/paper/1609.07843/) (2016)
- [Efficiently Modeling Long Sequences with Structured State Spaces](https://mlsanity.com/paper/2111.00396/) (2021)
- [Are Transformers Effective for Time Series Forecasting?](https://mlsanity.com/paper/2205.13504/) (2022)
- [Transformers are SSMs: Generalized Models and Efficient Algorithms Through Structured State Space Duality](https://mlsanity.com/paper/2405.21060/) (2024)
- [On the Properties of Neural Machine Translation: Encoder-Decoder Approaches](https://mlsanity.com/paper/1409.1259/) (2014)
- [Cross-lingual Language Model Pretraining](https://mlsanity.com/paper/1901.07291/) (2019)
- [Simplified State Space Layers for Sequence Modeling](https://mlsanity.com/paper/2208.04933/) (2022)
- [Long Range Arena: A Benchmark for Efficient Transformers](https://mlsanity.com/paper/2011.04006/) (2020)
- [HiPPO: Recurrent Memory with Optimal Polynomial Projections](https://mlsanity.com/paper/2008.07669/) (2020)
- [Gated Delta Networks: Improving Mamba2 with Delta Rule](https://mlsanity.com/paper/2412.06464/) (2024)
- [Gated Linear Attention Transformers with Hardware-Efficient Training](https://mlsanity.com/paper/2312.06635/) (2023)
- [Linear Transformers Are Secretly Fast Weight Programmers](https://mlsanity.com/paper/2102.11174/) (2021)
- [Resurrecting Recurrent Neural Networks for Long Sequences](https://mlsanity.com/paper/2303.06349/) (2023)
- [Parallelizing Linear Transformers with the Delta Rule over Sequence Length](https://mlsanity.com/paper/2406.06484/) (2024)
- [Transformers in Medical Imaging: A Survey](https://mlsanity.com/paper/2201.09873/) (2022)
- [Griffin: Mixing Gated Linear Recurrences with Local Attention for Efficient Language Models](https://mlsanity.com/paper/2402.19427/) (2024)
- [A Survey on Deep Neural Network Pruning-Taxonomy, Comparison, Analysis, and Recommendations](https://mlsanity.com/paper/2308.06767/) (2023)
- [Parallelizing Linear Recurrent Neural Nets Over Sequence Length](https://mlsanity.com/paper/1709.04057/) (2017)
- [RecurrentGemma: Moving Past Transformers for Efficient Open Language Models](https://mlsanity.com/paper/2404.07839/) (2024)
- [Spatial-Mamba: Effective Visual State Space Models via Structure-aware State Fusion](https://mlsanity.com/paper/2410.15091/) (2024)
- [Never Train from Scratch: Fair Comparison of Long-Sequence Models Requires Data-Driven Priors](https://mlsanity.com/paper/2310.02980/) (2023)
- [StableSSM: Alleviating the Curse of Memory in State-space Models through Stable Reparameterization](https://mlsanity.com/paper/2311.14495/) (2023)
- [Spectral State Space Models](https://mlsanity.com/paper/2312.06837/) (2023)
- [ReMamba: Equip Mamba with Effective Long-Sequence Modeling](https://mlsanity.com/paper/2408.15496/) (2024)
- [HOPE for a Robust Parameterization of Long-memory State Space Models](https://mlsanity.com/paper/2405.13975/) (2024)

## Built on by (most cited first, 1 of 1)

- [A Theoretical Analysis of Mamba's Training Dynamics: Filtering Relevant Features for Generalization in State Space Models](https://mlsanity.com/paper/2602.12499/) (2026)

## Abstract (arXiv)

State space models (SSMs) are an emerging paradigm that achieves linear-time scaling, however, they intrinsically suffer from "curse of memory". The memory of SSMs, including Mamba, decays exponentially as long as the recursive update is stable. In this work, we developed Long-range Integrative State space Models (LISM) to mitigate the curse of memory in SSMs. LISM allows the recursive update to occasionally be unstable, which enables longer memory retention, and incorporates a dynamic input-state interaction mechanism to identify inputs that should be memorized for longer periods. We theoretically establish probability bounds for the overall stability of LISM and derive a probabilistic stability condition to guide parameterization and training. We evaluated LISM's performance on multiple tasks, a synthetic Selective Copy task, two real language tasks WikiText-103 and LRA Retrieval, and an additional biological prediction task. LISM achieved the best performance on the Selective Copying task, WikiText-103, and the biological prediction task compared to SSM-based models and achieved comparable performance on the LRA-Retrieval task with the state-of-the-art. Beyond average performance, LISM also shows strong repeatability across runs. In the two real language tasks, LISM reduced the relative performance variance (coefficient of variation) by an average of 33.3%, achieving a standard deviation of ±0.30% on WikiText-103 compared to ±0.46% for Mamba-2. The implementation of LISM is available at https://github.com/CM-Compbio-Group/LISMdev.