# Generalizable, real-time neural decoding with hybrid state-space models

arXiv: https://arxiv.org/abs/2506.05320 · HTML: https://mlsanity.com/paper/2506.05320/

## Key facts

- Cited by 9 papers in the map (data through 14 Oct 2026).
- Builds on 39 of its 78 references that are in the map.
- First posted on arXiv 5 Jun 2025.

## Builds on (most cited first, 30 of 39)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [LoRA: Low-Rank Adaptation of Large Language Models](https://mlsanity.com/paper/2106.09685/) (2021)
- [Mamba: Linear-Time Sequence Modeling with Selective State Spaces](https://mlsanity.com/paper/2312.00752/) (2023)
- [RoFormer: Enhanced Transformer with Rotary Position Embedding](https://mlsanity.com/paper/2104.09864/) (2021)
- [Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translation](https://mlsanity.com/paper/1406.1078/) (2014)
- [Efficiently Modeling Long Sequences with Structured State Spaces](https://mlsanity.com/paper/2111.00396/) (2021)
- [Transformers are SSMs: Generalized Models and Efficient Algorithms Through Structured State Space Duality](https://mlsanity.com/paper/2405.21060/) (2024)
- [Perceiver: General Perception with Iterative Attention](https://mlsanity.com/paper/2103.03206/) (2021)
- [Simplified State Space Layers for Sequence Modeling](https://mlsanity.com/paper/2208.04933/) (2022)
- [Large Batch Optimization for Deep Learning: Training BERT in 76 minutes](https://mlsanity.com/paper/1904.00962/) (2019)
- [Perceiver IO: A General Architecture for Structured Inputs & Outputs](https://mlsanity.com/paper/2107.14795/) (2021)
- [Jamba: A Hybrid Transformer-Mamba Language Model](https://mlsanity.com/paper/2403.19887/) (2024)
- [Large Brain Model for Learning Generic Representations with Tremendous EEG Data in BCI](https://mlsanity.com/paper/2405.18765/) (2024)
- [Griffin: Mixing Gated Linear Recurrences with Local Attention for Efficient Language Models](https://mlsanity.com/paper/2402.19427/) (2024)
- [How to Train Your HiPPO: State Space Models with Generalized Orthogonal Basis Projections](https://mlsanity.com/paper/2206.12037/) (2022)
- [Learnable latent embeddings for joint behavioral and neural analysis](https://mlsanity.com/paper/2204.00673/) (2022)
- [Block-Recurrent Transformers](https://mlsanity.com/paper/2203.07852/) (2022)
- [BrainBERT: Self-supervised representation learning for intracranial recordings](https://mlsanity.com/paper/2302.14367/) (2023)
- [A Unified, Scalable Framework for Neural Population Decoding](https://mlsanity.com/paper/2310.16046/) (2023)
- [Neural Latents Benchmark '21: Evaluating latent variable models of neural population activity](https://mlsanity.com/paper/2109.04463/) (2021)
- [Representation learning for neural population activity with Neural Data Transformers](https://mlsanity.com/paper/2108.01210/) (2021)
- [Machine learning for neural decoding](https://mlsanity.com/paper/1708.00909/) (2017)
- [Mine Your Own vieW: Self-Supervised Learning Through Across-Sample Prediction](https://mlsanity.com/paper/2102.10106/) (2021)
- [Large-scale neural recordings call for new insights to link brain and behavior](https://mlsanity.com/paper/2103.14662/) (2021)
- [Towards a "universal translator" for neural dynamics at single-cell, single-spike resolution](https://mlsanity.com/paper/2407.14668/) (2024)
- [Adversarial Domain Adaptation for Stable Brain-Machine Interfaces](https://mlsanity.com/paper/1810.00045/) (2018)
- [Population Transformer: Learning Population-level Representations of Neural Activity](https://mlsanity.com/paper/2406.03044/) (2024)
- [Block-State Transformers](https://mlsanity.com/paper/2306.09539/) (2023)
- [Making brain-machine interfaces robust to future neural variability](https://mlsanity.com/paper/1610.05872/) (2016)

## Built on by (most cited first, 9 of 9)

- [SPINT: Spatial Permutation-Invariant Neural Transformer for Consistent Intracortical Motor Decoding](https://mlsanity.com/paper/2507.08402/) (2025)
- [A cross-species neural foundation model for end-to-end speech decoding](https://mlsanity.com/paper/2511.21740/) (2025)
- [Geometry-aware similarity metrics for neural representations on Riemannian and statistical manifolds](https://mlsanity.com/paper/2603.28764/) (2026)
- [Benchmarking Probabilistic Time Series Forecasting Models on Neural Activity](https://mlsanity.com/paper/2510.18037/) (2025)
- [RPNT: Robust Pre-trained Neural Transformer -- A Pathway for Generalized Motor Decoding](https://mlsanity.com/paper/2601.17641/) (2026)
- [JEDI: Jointly Embedded Inference of Neural Dynamics](https://mlsanity.com/paper/2603.10489/) (2026)
- [Machine Learning Methods for Studying Latent Neural Activity Dynamics](https://mlsanity.com/paper/2606.10530/) (2026)
- [Real-time Reconstruction of Human Visual Perception from fMRI](https://mlsanity.com/paper/2607.22753/) (2026)
- [NeuroLens: Learning Latent Embeddings of Neural Semantics from Chronic Recordings](https://mlsanity.com/paper/2610.02864/) (2026)

## Abstract (arXiv)

Real-time decoding of neural activity is central to neuroscience and neurotechnology applications, from closed-loop experiments to brain-computer interfaces, where models are subject to strict latency constraints. Traditional methods, including simple recurrent neural networks, are fast and lightweight but often struggle to generalize to unseen data. In contrast, recent Transformer-based approaches leverage large-scale pretraining for strong generalization performance, but typically have much larger computational requirements and are not always suitable for low-resource or real-time settings. To address these shortcomings, we present POSSM, a novel hybrid architecture that combines individual spike tokenization via a cross-attention module with a recurrent state-space model (SSM) backbone to enable (1) fast and causal online prediction on neural activity and (2) efficient generalization to new sessions, individuals, and tasks through multi-dataset pretraining. We evaluate POSSM's decoding performance and inference speed on intracortical decoding of monkey motor tasks, and show that it extends to clinical applications, namely handwriting and speech decoding in human subjects. Notably, we demonstrate that pretraining on monkey motor-cortical recordings improves decoding performance on the human handwriting task, highlighting the exciting potential for cross-species transfer. In all of these tasks, we find that POSSM achieves decoding accuracy comparable to state-of-the-art Transformers, at a fraction of the inference cost (up to 9x faster on GPU). These results suggest that hybrid SSMs are a promising approach to bridging the gap between accuracy, inference speed, and generalization when training neural decoders for real-time, closed-loop applications.