# Structured Linear CDEs: Maximally Expressive and Parallel-in-Time Sequence Models

arXiv: https://arxiv.org/abs/2505.17761 · HTML: https://mlsanity.com/paper/2505.17761/

## Key facts

- Cited by 18 papers in the map (data through 14 Oct 2026).
- Builds on 67 of its 111 references that are in the map.
- First posted on arXiv 23 May 2025.

## Builds on (most cited first, 30 of 67)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [Layer Normalization](https://mlsanity.com/paper/1607.06450/) (2016)
- [Mamba: Linear-Time Sequence Modeling with Selective State Spaces](https://mlsanity.com/paper/2312.00752/) (2023)
- [Efficiently Modeling Long Sequences with Structured State Spaces](https://mlsanity.com/paper/2111.00396/) (2021)
- [The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networks](https://mlsanity.com/paper/1803.03635/) (2018)
- [Transformers are RNNs: Fast Autoregressive Transformers with Linear Attention](https://mlsanity.com/paper/2006.16236/) (2020)
- [Generating Long Sequences with Sparse Transformers](https://mlsanity.com/paper/1904.10509/) (2019)
- [Transformers are SSMs: Generalized Models and Efficient Algorithms Through Structured State Space Duality](https://mlsanity.com/paper/2405.21060/) (2024)
- [Simplified State Space Layers for Sequence Modeling](https://mlsanity.com/paper/2208.04933/) (2022)
- [Language Modeling with Gated Convolutional Networks](https://mlsanity.com/paper/1612.08083/) (2016)
- [Channel Pruning for Accelerating Very Deep Neural Networks](https://mlsanity.com/paper/1707.06168/) (2017)
- [Gated Delta Networks: Improving Mamba2 with Delta Rule](https://mlsanity.com/paper/2412.06464/) (2024)
- [Gated Linear Attention Transformers with Hardware-Efficient Training](https://mlsanity.com/paper/2312.06635/) (2023)
- [Neural Controlled Differential Equations for Irregular Time Series](https://mlsanity.com/paper/2005.08926/) (2020)
- [Linear Transformers Are Secretly Fast Weight Programmers](https://mlsanity.com/paper/2102.11174/) (2021)
- [Pruning neural networks without any data by iteratively conserving synaptic flow](https://mlsanity.com/paper/2006.05467/) (2020)
- [On the Parameterization and Initialization of Diagonal State Space Models](https://mlsanity.com/paper/2206.11893/) (2022)
- [Resurrecting Recurrent Neural Networks for Long Sequences](https://mlsanity.com/paper/2303.06349/) (2023)
- [Parallelizing Linear Transformers with the Delta Rule over Sequence Length](https://mlsanity.com/paper/2406.06484/) (2024)
- [Learning to (Learn at Test Time): RNNs with Expressive Hidden States](https://mlsanity.com/paper/2407.04620/) (2024)
- [Random Feature Attention](https://mlsanity.com/paper/2103.02143/) (2021)
- [The UEA multivariate time series classification archive, 2018](https://mlsanity.com/paper/1811.00075/) (2018)
- [Titans: Learning to Memorize at Test Time](https://mlsanity.com/paper/2501.00663/) (2024)
- [xLSTM: Extended Long Short-Term Memory](https://mlsanity.com/paper/2405.04517/) (2024)
- [On Neural Differential Equations](https://mlsanity.com/paper/2202.02435/) (2022)
- [Transformers Learn Shortcuts to Automata](https://mlsanity.com/paper/2210.10749/) (2022)
- [Accelerating Sparse Deep Neural Networks](https://mlsanity.com/paper/2104.08378/) (2021)
- [Neural Networks and the Chomsky Hierarchy](https://mlsanity.com/paper/2207.02098/) (2022)
- [The Parallelism Tradeoff: Limitations of Log-Precision Transformers](https://mlsanity.com/paper/2207.00729/) (2022)

## Built on by (most cited first, 18 of 18)

- [Structured Sparse Transition Matrices to Enable State Tracking in State-Space Models](https://mlsanity.com/paper/2509.22284/) (2025)
- [pySigLib -- Fast Signature-Based Computations on CPU and GPU](https://mlsanity.com/paper/2509.10613/) (2025)
- [Learning State-Tracking from Code Using Linear RNNs](https://mlsanity.com/paper/2602.14814/) (2026)
- [The Curious Case of In-Training Compression of State Space Models](https://mlsanity.com/paper/2510.02823/) (2025)
- [Neural CDEs as Correctors for Learned Time Series Models](https://mlsanity.com/paper/2512.12116/) (2025)
- [Efficient Neural Controlled Differential Equations via Attentive Kernel Smoothing](https://mlsanity.com/paper/2602.02157/) (2026)
- [Why Depth Matters in Parallelizable Sequence Models: A Lie Algebraic View](https://mlsanity.com/paper/2603.05573/) (2026)
- [Length Generalization with Log-Depth Recurrent Units](https://mlsanity.com/paper/2605.26035/) (2026)
- [Chess-World-Model: A 10M-Game Benchmark for Exact State Tracking from Chess Move Sequences](https://mlsanity.com/paper/2605.30100/) (2026)
- [SHiPPO: Recurrent Memory with Transported Polynomial Projections](https://mlsanity.com/paper/2607.03055/) (2026)
- [Novelty detection on path space](https://mlsanity.com/paper/2512.03243/) (2025)
- [Improved state mixing in higher-order and block diagonal linear recurrent networks](https://mlsanity.com/paper/2602.12021/) (2026)
- [Extending SSMs with the Exponentially Weighted Signature](https://mlsanity.com/paper/2603.19198/) (2026)
- [Flash PD-SSM: Memory-Optimized Structured Sparse State-Space Models](https://mlsanity.com/paper/2605.19150/) (2026)
- [Universal Time Series Generation with Neural Controlled Differential Equations](https://mlsanity.com/paper/2605.28507/) (2026)
- [Faithful Embeddings of Irregular and Asynchronous Data for Online Log-NCDEs](https://mlsanity.com/paper/2605.30213/) (2026)
- [Structured Neural SDEs for Functional Calibration](https://mlsanity.com/paper/2609.34831/) (2026)
- [LogSig-SSM: Time-Series Modelling with Multi-Scale Log-Signature Compression for State-Space Models](https://mlsanity.com/paper/2610.05051/) (2026)

## Abstract (arXiv)

This work introduces Structured Linear Controlled Differential Equations (SLiCEs), a unifying framework for sequence models with structured, input-dependent state-transition matrices that retain the maximal expressivity of dense matrices whilst being cheaper to compute. The framework encompasses existing architectures, such as input-dependent block-diagonal linear recurrent neural networks and DeltaNet's diagonal-plus-low-rank structure, as well as two novel variants based on sparsity and the Walsh-Hadamard transform. We prove that, unlike the diagonal state-transition matrices of S4D and Mamba, SLiCEs employing block-diagonal, sparse, or Walsh-Hadamard matrices match the maximal expressivity of dense matrices. Empirically, SLiCEs solve the A5 state-tracking benchmark with a single layer, achieve best-in-class length generalisation on regular language tasks among parallel-in-time models, and match the performance of log neural controlled differential equations on six multivariate time-series classification datasets while cutting the average time per training step by a factor of twenty.