# Effectively Modeling Time Series with Simple Discrete State Spaces

arXiv: https://arxiv.org/abs/2303.09489 · HTML: https://mlsanity.com/paper/2303.09489/

## Key facts

- Cited by 54 papers in the map (data through 14 Oct 2026).
- Builds on 39 of its 86 references that are in the map.
- First posted on arXiv 16 Mar 2023.

## Builds on (most cited first, 30 of 39)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Neural Ordinary Differential Equations](https://mlsanity.com/paper/1806.07366/) (2018)
- [Longformer: The Long-Document Transformer](https://mlsanity.com/paper/2004.05150/) (2020)
- [Efficiently Modeling Long Sequences with Structured State Spaces](https://mlsanity.com/paper/2111.00396/) (2021)
- [Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modeling](https://mlsanity.com/paper/1412.3555/) (2014)
- [Informer: Beyond Efficient Transformer for Long Sequence Time-Series Forecasting](https://mlsanity.com/paper/2012.07436/) (2020)
- [Conformer: Convolution-augmented Transformer for Speech Recognition](https://mlsanity.com/paper/2005.08100/) (2020)
- [WaveNet: A Generative Model for Raw Audio](https://mlsanity.com/paper/1609.03499/) (2016)
- [Autoformer: Decomposition Transformers with Auto-Correlation for Long-Term Series Forecasting](https://mlsanity.com/paper/2106.13008/) (2021)
- [Reformer: The Efficient Transformer](https://mlsanity.com/paper/2001.04451/) (2020)
- [Rethinking Attention with Performers](https://mlsanity.com/paper/2009.14794/) (2020)
- [Are Transformers Effective for Time Series Forecasting?](https://mlsanity.com/paper/2205.13504/) (2022)
- [FEDformer: Frequency Enhanced Decomposed Transformer for Long-term Series Forecasting](https://mlsanity.com/paper/2201.12740/) (2022)
- [DeepAR: Probabilistic Forecasting with Autoregressive Recurrent Networks](https://mlsanity.com/paper/1704.04110/) (2017)
- [Simplified State Space Layers for Sequence Modeling](https://mlsanity.com/paper/2208.04933/) (2022)
- [N-BEATS: Neural basis expansion analysis for interpretable time series forecasting](https://mlsanity.com/paper/1905.10437/) (2019)
- [Speech Commands: A Dataset for Limited-Vocabulary Speech Recognition](https://mlsanity.com/paper/1804.03209/) (2018)
- [Long Range Arena: A Benchmark for Efficient Transformers](https://mlsanity.com/paper/2011.04006/) (2020)
- [Combining Recurrent, Convolutional, and Continuous-time Models with Linear State-Space Layers](https://mlsanity.com/paper/2110.13985/) (2021)
- [Neural Controlled Differential Equations for Irregular Time Series](https://mlsanity.com/paper/2005.08926/) (2020)
- [On the Parameterization and Initialization of Diagonal State Space Models](https://mlsanity.com/paper/2206.11893/) (2022)
- [Diagonal State Spaces are as Effective as Structured State Spaces](https://mlsanity.com/paper/2203.14343/) (2022)
- [On Neural Differential Equations](https://mlsanity.com/paper/2202.02435/) (2022)
- [Monash Time Series Forecasting Archive](https://mlsanity.com/paper/2105.06643/) (2021)
- [Adversarial Audio Synthesis](https://mlsanity.com/paper/1802.04208/) (2018)
- [CatBoost: gradient boosting with categorical features support](https://mlsanity.com/paper/1810.11363/) (2018)
- [Diffusion-based Time Series Imputation and Forecasting with Structured State Space Models](https://mlsanity.com/paper/2208.09399/) (2022)
- [It's Raw! Audio Generation with State-Space Models](https://mlsanity.com/paper/2202.09729/) (2022)
- [Automatic diagnosis of the 12-lead ECG using a deep neural network](https://mlsanity.com/paper/1904.01949/) (2019)
- [Deep Learning for ECG Analysis: Benchmarks and Insights from PTB-XL](https://mlsanity.com/paper/2004.13701/) (2020)

## Built on by (most cited first, 30 of 54)

- [Graph Mamba: Towards Learning on Graphs with State Space Models](https://mlsanity.com/paper/2402.08678/) (2024)
- [Mechanistic Design and Scaling of Hybrid Architectures](https://mlsanity.com/paper/2403.17844/) (2024)
- [MambaMixer: Efficient Selective State Space Models with Dual Token and Channel Selection](https://mlsanity.com/paper/2403.19888/) (2024)
- [FlashFFTConv: Efficient Convolutions for Long Sequences with Tensor Cores](https://mlsanity.com/paper/2311.05908/) (2023)
- [Universal Time-Series Representation Learning: A Survey](https://mlsanity.com/paper/2401.03717/) (2024)
- [Laughing Hyena Distillery: Extracting Compact Recurrences From Convolutions](https://mlsanity.com/paper/2310.18780/) (2023)
- [Spectral State Space Models](https://mlsanity.com/paper/2312.06837/) (2023)
- [Generative AI for End-to-End Limit Order Book Modelling: A Token-Level Autoregressive Generative Model of Message Flow Using a Deep State Space Network](https://mlsanity.com/paper/2309.00638/) (2023)
- [Sequence Modeling with Multiresolution Convolutional Memory](https://mlsanity.com/paper/2305.01638/) (2023)
- [Audio Mamba: Selective State Spaces for Self-Supervised Audio Representations](https://mlsanity.com/paper/2406.02178/) (2024)
- [State-Free Inference of State-Space Models: The Transfer Function Approach](https://mlsanity.com/paper/2405.06147/) (2024)
- [RGBT Tracking via All-layer Multimodal Interactions with Progressive Fusion Mamba](https://mlsanity.com/paper/2408.08827/) (2024)
- [State Space Models on Temporal Graphs: A First-Principles Study](https://mlsanity.com/paper/2406.00943/) (2024)
- [Bi-Mamba+: Bidirectional Mamba for Time Series Forecasting](https://mlsanity.com/paper/2404.15772/) (2024)
- [MambaTS: Improved Selective State Space Models for Long-term Time Series Forecasting](https://mlsanity.com/paper/2405.16440/) (2024)
- [xLSTMTime : Long-term Time Series Forecasting With xLSTM](https://mlsanity.com/paper/2407.10240/) (2024)
- [Focus Your Attention (with Adaptive IIR Filters)](https://mlsanity.com/paper/2305.14952/) (2023)
- [On the Long Range Abilities of Transformers](https://mlsanity.com/paper/2311.16620/) (2023)
- [Layer-Adaptive State Pruning for Deep State Space Models](https://mlsanity.com/paper/2411.02824/) (2024)
- [Time-SSM: Simplifying and Unifying State Space Models for Time Series Forecasting](https://mlsanity.com/paper/2405.16312/) (2024)
- [Chimera: Effectively Modeling Multivariate Time Series with 2-Dimensional State Space Models](https://mlsanity.com/paper/2406.04320/) (2024)
- [Scaling Representation Learning from Ubiquitous ECG with State-Space Models](https://mlsanity.com/paper/2309.15292/) (2023)
- [Multispectral State-Space Feature Fusion: Bridging Shared and Cross-Parametric Interactions for Object Detection](https://mlsanity.com/paper/2507.14643/) (2025)
- [The Seismic Wavefield Common Task Framework](https://mlsanity.com/paper/2512.19927/) (2025)
- [Sequential Order-Robust Mamba for Time Series Forecasting](https://mlsanity.com/paper/2410.23356/) (2024)
- [Enhancing Multivariate Time Series Forecasting with Global Temporal Retrieval](https://mlsanity.com/paper/2602.10847/) (2026)
- [CTF4Nuclear: Common Task Framework for Nuclear Fission and Fusion Models](https://mlsanity.com/paper/2605.15549/) (2026)
- [Towards trustworthy seizure onset detection using workflow notes](https://mlsanity.com/paper/2306.08728/) (2023)
- [Identity Curvature Laplace Approximation for Improved Out-of-Distribution Detection](https://mlsanity.com/paper/2312.10464/) (2023)
- [Hypergraph Neural Network with State Space Models for Node Classification](https://mlsanity.com/paper/2508.06587/) (2025)

## Abstract (arXiv)

Time series modeling is a well-established problem, which often requires that methods (1) expressively represent complicated dependencies, (2) forecast long horizons, and (3) efficiently train over long sequences. State-space models (SSMs) are classical models for time series, and prior works combine SSMs with deep learning layers for efficient sequence modeling. However, we find fundamental limitations with these prior approaches, proving their SSM representations cannot express autoregressive time series processes. We thus introduce SpaceTime, a new state-space time series architecture that improves all three criteria. For expressivity, we propose a new SSM parameterization based on the companion matrix -- a canonical representation for discrete-time processes -- which enables SpaceTime's SSM layers to learn desirable autoregressive processes. For long horizon forecasting, we introduce a "closed-loop" variation of the companion SSM, which enables SpaceTime to predict many future time-steps by generating its own layer-wise inputs. For efficient training and inference, we introduce an algorithm that reduces the memory and compute of a forward pass with the companion matrix. With sequence length ℓ and state-space size d, we go from tildeO(d ℓ) na"ively to tildeO(d + ℓ). In experiments, our contributions lead to state-of-the-art results on extensive and diverse benchmarks, with best or second-best AUROC on 6 / 7 ECG and speech time series classification, and best MSE on 14 / 16 Informer forecasting tasks. Furthermore, we find SpaceTime (1) fits AR(p) processes that prior deep SSMs fail on, (2) forecasts notably more accurately on longer horizons than prior state-of-the-art, and (3) speeds up training on real-world ETTh1 data by 73% and 80% relative wall-clock time over Transformers and LSTMs.