# Context-Selective State Space Models: Feedback is All You Need

arXiv: https://arxiv.org/abs/2510.14027 · HTML: https://mlsanity.com/paper/2510.14027/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Placed in Core SSM architectures for language, one of 10 directions in State-space sequence models (S4, Mamba).
- Builds on 36 of its 50 references that are in the map.
- First posted on arXiv 15 Oct 2025.

## Placed in

- State-space sequence models (S4, Mamba) / [Core SSM architectures for language](https://mlsanity.com/ssm/core-ssm-architectures-for-language/)

## Builds on (most cited first, 30 of 36)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning](https://mlsanity.com/paper/2501.12948/) (2025)
- [Mamba: Linear-Time Sequence Modeling with Selective State Spaces](https://mlsanity.com/paper/2312.00752/) (2023)
- [Gaussian Error Linear Units (GELUs)](https://mlsanity.com/paper/1606.08415/) (2016)
- [Longformer: The Long-Document Transformer](https://mlsanity.com/paper/2004.05150/) (2020)
- [Efficiently Modeling Long Sequences with Structured State Spaces](https://mlsanity.com/paper/2111.00396/) (2021)
- [Big Bird: Transformers for Longer Sequences](https://mlsanity.com/paper/2007.14062/) (2020)
- [Reformer: The Efficient Transformer](https://mlsanity.com/paper/2001.04451/) (2020)
- [Rethinking Attention with Performers](https://mlsanity.com/paper/2009.14794/) (2020)
- [Transformers are SSMs: Generalized Models and Efficient Algorithms Through Structured State Space Duality](https://mlsanity.com/paper/2405.21060/) (2024)
- [In-context Learning and Induction Heads](https://mlsanity.com/paper/2209.11895/) (2022)
- [Simplified State Space Layers for Sequence Modeling](https://mlsanity.com/paper/2208.04933/) (2022)
- [HiPPO: Recurrent Memory with Optimal Polynomial Projections](https://mlsanity.com/paper/2008.07669/) (2020)
- [Resurrecting Recurrent Neural Networks for Long Sequences](https://mlsanity.com/paper/2303.06349/) (2023)
- [Axial Attention in Multidimensional Transformers](https://mlsanity.com/paper/1912.12180/) (2019)
- [Diagonal State Spaces are as Effective as Structured State Spaces](https://mlsanity.com/paper/2203.14343/) (2022)
- [A mathematical perspective on Transformers](https://mlsanity.com/paper/2312.10794/) (2023)
- [Liquid Structural State-Space Models](https://mlsanity.com/paper/2209.12951/) (2022)
- [Theoretical Foundations of Deep Selective State-Space Models](https://mlsanity.com/paper/2402.19047/) (2024)
- [Parallelizing non-linear sequential models over the sequence length](https://mlsanity.com/paper/2309.12252/) (2023)
- [Oscillatory State-Space Models](https://mlsanity.com/paper/2410.03943/) (2024)
- [Towards Scalable and Stable Parallelization of Nonlinear RNNs](https://mlsanity.com/paper/2407.19115/) (2024)
- [Temporal Parallelization of Bayesian Smoothers](https://mlsanity.com/paper/1905.13002/) (2019)
- [B'MOJO: Hybrid State Space Realizations of Foundation Models with Eidetic and Fading Memory](https://mlsanity.com/paper/2407.06324/) (2024)
- [Heat Death of Generative Models in Closed-Loop Learning](https://mlsanity.com/paper/2404.02325/) (2024)
- [EdgeFormer: A Parameter-Efficient Transformer for On-Device Seq2seq Generation](https://mlsanity.com/paper/2202.07959/) (2022)
- [State Space Models as Foundation Models: A Control Theoretic Overview](https://mlsanity.com/paper/2403.16899/) (2024)
- [State-Free Inference of State-Space Models: The Transfer Function Approach](https://mlsanity.com/paper/2405.06147/) (2024)

## Built on by (most cited first, 2 of 2)

- [Switching Linear Attention](https://mlsanity.com/paper/2609.39034/) (2026)
- [Beyond State Capacity: Why Retrieval Breaks in Mamba](https://mlsanity.com/paper/2610.19709/) (2026)

## Abstract (arXiv)

Transformers, powered by the attention mechanism, are the backbone of most foundation models, yet they suffer from quadratic complexity and difficulties in dealing with long-range dependencies in the input sequence. Recent work has shown that state space models (SSMs) provide a promising alternative. In this paper, we introduce the COFFEE (COntext From FEEdback) model, a novel time-varying SSM that incorporates state feedback to enable context-dependent selectivity, while still allowing for parallel implementation. This idea allows the model to regulate its dynamics based on the context described by the internal state, which embodies a compact representation of the input history. State feedback allows COFFEE to improve its ability to capture long-range dependencies: on the induction head task, it achieves near-perfect accuracy with two orders of magnitude fewer parameters and training sequences compared to S6 (the SSM of Mamba). On MNIST, COFFEE largely outperforms S6 within the same architecture, reaching 97% accuracy with only 3585 parameters. These results showcase the role of state feedback as a key mechanism for building scalable and efficient sequence models.