mlsanityData to 14 Oct 2026

Context-Selective State Space Models: Feedback is All You Need

arXiv 2510.14027 ↗First posted 15 Oct 2025cs.LG cs.AIcited by 2

On the map: State-space sequence models (S4, Mamba) → Core SSM architectures for language

Abstract (arXiv)

Transformers, powered by the attention mechanism, are the backbone of most foundation models, yet they suffer from quadratic complexity and difficulties in dealing with long-range dependencies in the input sequence. Recent work has shown that state space models (SSMs) provide a promising alternative. In this paper, we introduce the COFFEE (COntext From FEEdback) model, a novel time-varying SSM that incorporates state feedback to enable context-dependent selectivity, while still allowing for parallel implementation. This idea allows the model to regulate its dynamics based on the context described by the internal state, which embodies a compact representation of the input history. State feedback allows COFFEE to improve its ability to capture long-range dependencies: on the induction head task, it achieves near-perfect accuracy with two orders of magnitude fewer parameters and training sequences compared to S6 (the SSM of Mamba). On MNIST, COFFEE largely outperforms S6 within the same architecture, reaching 97% accuracy with only 3585 parameters. These results showcase the role of state feedback as a key mechanism for building scalable and efficient sequence models.

Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Placed in Core SSM architectures for language, one of 10 directions in State-space sequence models (S4, Mamba). Builds on 36 of its 50 references that are in the map. First posted on arXiv 15 Oct 2025.

Builds on

36 of 50 references.

Closest ancestors · 8

Same direction, newest first

  1. 2025Parallelization of Non-linear State-Space Models: Scaling Up Liquid-Resistance Liquid-Capacitance Networks for Efficient Sequence Modeling27 May
  2. 2024Slot State Space Models18 Jun
  3. 2024Transformers are SSMs: Generalized Models and Efficient Algorithms Through Structured State Space Duality31 May
  4. 2024Theoretical Foundations of Deep Selective State-Space Models29 Feb
  5. 2023Mamba: Linear-Time Sequence Modeling with Selective State Spaces1 Dec
  6. 2022Simplified State Space Layers for Sequence Modeling9 Aug
  7. 2022Diagonal State Spaces are as Effective as Structured State Spaces27 Mar
  8. 2021Efficiently Modeling Long Sequences with Structured State Spaces31 Oct

Other references · 28

  1. 2017Attention Is All You Needcited by 60,361
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  3. 2023GPT-4 Technical Reportcited by 18,396
  4. 2025DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learningcited by 6,330
  5. 2016Gaussian Error Linear Units (GELUs)cited by 3,502
  6. 2020Longformer: The Long-Document Transformercited by 3,413
  7. 2020Big Bird: Transformers for Longer Sequencescited by 1,684
  8. 2020Reformer: The Efficient Transformercited by 1,668
  9. 2020Rethinking Attention with Performerscited by 1,601
  10. 2022In-context Learning and Induction Headscited by 964
  11. 2020HiPPO: Recurrent Memory with Optimal Polynomial Projectionscited by 605
  12. 2023Resurrecting Recurrent Neural Networks for Long Sequencescited by 408
16 more
  1. 2019Axial Attention in Multidimensional Transformerscited by 358
  2. 2023A mathematical perspective on Transformerscited by 164
  3. 2022Liquid Structural State-Space Modelscited by 136
  4. 2023Parallelizing non-linear sequential models over the sequence lengthcited by 45
  5. 2024Oscillatory State-Space Modelscited by 39
  6. 2024Towards Scalable and Stable Parallelization of Nonlinear RNNscited by 38
  7. 2019Temporal Parallelization of Bayesian Smootherscited by 33
  8. 2024B'MOJO: Hybrid State Space Realizations of Foundation Models with Eidetic and Fading Memorycited by 26
  9. 2024Heat Death of Generative Models in Closed-Loop Learningcited by 22
  10. 2022EdgeFormer: A Parameter-Efficient Transformer for On-Device Seq2seq Generationcited by 21
  11. 2024State Space Models as Foundation Models: A Control Theoretic Overviewcited by 19
  12. 2024State-Free Inference of State-Space Models: The Transfer Function Approachcited by 18
  13. 2024The Asymptotic Behavior of Attention in Transformerscited by 16
  14. 2024State Space Models are Provably Comparable to Transformers in Dynamic Token Selectioncited by 6
  15. 2018ProdSumNet: reducing model parameters in deep neural networks via product-of-sums matrix decompositionscited by 5
  16. 2024Meanings and Feelings of Large Language Models: Observability of Latent States in Generative AIcited by 5

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 2.

2 papers.

  1. 2026Switching Linear Attentioncited by 0
  2. 2026Beyond State Capacity: Why Retrieval Breaks in Mambacited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024S7: Selective and Simplified State Space Layers for Sequence Modeling
  2. 2025Rethinking Selectivity in State Space Models: A Minimal Predictive Sufficiency Approach
  3. 2025LISM: Long-range Integrative State space Models via Input-Latent State Interactions
  4. 2026Interpreting and Steering State-Space Models via Activation Subspace Bottlenecks
  5. 2025Advancing Intelligent Sequence Modeling: Evolution, Trade-offs, and Applications of State-Space Architectures from S4 to Mamba
  6. 2024Understanding and Mitigating Bottlenecks of State Space Models through the Lens of Recency and Over-smoothing
  7. 2025Controllability Analysis of State Space-based Language Model
  8. 2025SeRpEnt: Selective Resampling for Expressive State Space Models