mlsanityData to 14 Oct 2026

Structured Linear CDEs: Maximally Expressive and Parallel-in-Time Sequence Models

arXiv 2505.17761 ↗First posted 23 May 2025cs.LGcited by 18

Not on the map. Why?

Abstract (arXiv)

This work introduces Structured Linear Controlled Differential Equations (SLiCEs), a unifying framework for sequence models with structured, input-dependent state-transition matrices that retain the maximal expressivity of dense matrices whilst being cheaper to compute. The framework encompasses existing architectures, such as input-dependent block-diagonal linear recurrent neural networks and DeltaNet's diagonal-plus-low-rank structure, as well as two novel variants based on sparsity and the Walsh-Hadamard transform. We prove that, unlike the diagonal state-transition matrices of S4D and Mamba, SLiCEs employing block-diagonal, sparse, or Walsh-Hadamard matrices match the maximal expressivity of dense matrices. Empirically, SLiCEs solve the A5 state-tracking benchmark with a single layer, achieve best-in-class length generalisation on regular language tasks among parallel-in-time models, and match the performance of log neural controlled differential equations on six multivariate time-series classification datasets while cutting the average time per training step by a factor of twenty.

Key facts. Cited by 18 papers in the map (data through 14 Oct 2026). Builds on 67 of its 111 references that are in the map. First posted on arXiv 23 May 2025.

Builds on

67 of 111 references.

References · 67

  1. 2017Attention Is All You Needcited by 60,361
  2. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,305
  3. 2016Layer Normalizationcited by 5,669
  4. 2023Mamba: Linear-Time Sequence Modeling with Selective State Spacescited by 4,718
  5. 2021Efficiently Modeling Long Sequences with Structured State Spacescited by 2,465
  6. 2018The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networkscited by 2,262
  7. 2020Transformers are RNNs: Fast Autoregressive Transformers with Linear Attentioncited by 2,022
  8. 2019Generating Long Sequences with Sparse Transformerscited by 1,546
  9. 2024Transformers are SSMs: Generalized Models and Efficient Algorithms Through Structured State Space Dualitycited by 1,336
  10. 2022Simplified State Space Layers for Sequence Modelingcited by 788
  11. 2016Language Modeling with Gated Convolutional Networkscited by 702
  12. 2017Channel Pruning for Accelerating Very Deep Neural Networkscited by 702
38 more
  1. 2024Gated Delta Networks: Improving Mamba2 with Delta Rulecited by 532
  2. 2023Gated Linear Attention Transformers with Hardware-Efficient Trainingcited by 514
  3. 2020Neural Controlled Differential Equations for Irregular Time Seriescited by 506
  4. 2021Linear Transformers Are Secretly Fast Weight Programmerscited by 502
  5. 2020Pruning neural networks without any data by iteratively conserving synaptic flowcited by 479
  6. 2022On the Parameterization and Initialization of Diagonal State Space Modelscited by 424
  7. 2023Resurrecting Recurrent Neural Networks for Long Sequencescited by 408
  8. 2024Parallelizing Linear Transformers with the Delta Rule over Sequence Lengthcited by 396
  9. 2024Learning to (Learn at Test Time): RNNs with Expressive Hidden Statescited by 355
  10. 2021Random Feature Attentioncited by 354
  11. 2018The UEA multivariate time series classification archive, 2018cited by 338
  12. 2024Titans: Learning to Memorize at Test Timecited by 328
  13. 2024xLSTM: Extended Long Short-Term Memorycited by 327
  14. 2022On Neural Differential Equationscited by 268
  15. 2022Transformers Learn Shortcuts to Automatacited by 268
  16. 2021Accelerating Sparse Deep Neural Networkscited by 223
  17. 2022Neural Networks and the Chomsky Hierarchycited by 217
  18. 2022The Parallelism Tradeoff: Limitations of Log-Precision Transformerscited by 193
  19. 2024The Illusion of State in State-Space Modelscited by 187
  20. 2020Neural Rough Differential Equations for Long Time Seriescited by 143
  21. 2025RWKV-7 "Goose" with Expressive Dynamic State Evolutioncited by 141
  22. 2017Parallelizing Linear Recurrent Neural Nets Over Sequence Lengthcited by 127
  23. 2024HGRN2: Gated Linear RNNs with State Expansioncited by 127
  24. 2022Monarch: Expressive Structured Matrices for Efficient and Accurate Trainingcited by 106
  25. 2016A Primer on the Signature Method in Machine Learningcited by 104
  26. 2024Unlocking State-Tracking in Linear RNNs Through Negative Eigenvaluescited by 101
  27. 2016Kernels for sequentially ordered datacited by 84
  28. 2024Gated Slot Attention for Efficient Linear-Time Sequence Modelingcited by 76
  29. 2019Deep Signature Transformscited by 75
  30. 2025DeltaProduct: Improving State-Tracking in Linear RNNs via Householder Productscited by 75
  31. 2024Theoretical Foundations of Deep Selective State-Space Modelscited by 74
  32. 2020The Signature Kernel is the solution of a Goursat PDEcited by 58
  33. 2023Parallelizing non-linear sequential models over the sequence lengthcited by 45
  34. 2024Towards Scalable and Stable Parallelization of Nonlinear RNNscited by 38
  35. 2023Non-adversarial training of Neural SDEs with signature kernel scorescited by 37
  36. 2024Log Neural Controlled Differential Equations: The Lie Brackets Make a Differencecited by 35
  37. 2020Distribution Regression for Sequential Datacited by 31
  38. 2021Higher Order Kernel Mean Embeddings to Capture Filtrations of Stochastic Processescited by 27

50 most cited of 67 All 67 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 5; 2026: 13.

18 papers.

Sort:

By area:

  1. 2025Structured Sparse Transition Matrices to Enable State Tracking in State-Space Modelscited by 16
  2. 2025pySigLib -- Fast Signature-Based Computations on CPU and GPUcited by 6
  3. 2026Learning State-Tracking from Code Using Linear RNNscited by 6
  4. 2025The Curious Case of In-Training Compression of State Space Modelscited by 4
  5. 2025Neural CDEs as Correctors for Learned Time Series Modelscited by 1
  6. 2026Efficient Neural Controlled Differential Equations via Attentive Kernel Smoothingcited by 1
  7. 2026Why Depth Matters in Parallelizable Sequence Models: A Lie Algebraic Viewcited by 1
  8. 2026Length Generalization with Log-Depth Recurrent Unitscited by 1
  9. 2026Chess-World-Model: A 10M-Game Benchmark for Exact State Tracking from Chess Move Sequencescited by 1
  10. 2026SHiPPO: Recurrent Memory with Transported Polynomial Projectionscited by 1
  11. 2025Novelty detection on path spacecited by 0
  12. 2026Improved state mixing in higher-order and block diagonal linear recurrent networkscited by 0
6 more
  1. 2026Extending SSMs with the Exponentially Weighted Signaturecited by 0
  2. 2026Flash PD-SSM: Memory-Optimized Structured Sparse State-Space Modelscited by 0
  3. 2026Universal Time Series Generation with Neural Controlled Differential Equationscited by 0
  4. 2026Faithful Embeddings of Irregular and Asynchronous Data for Online Log-NCDEscited by 0
  5. 2026Structured Neural SDEs for Functional Calibrationcited by 0
  6. 2026LogSig-SSM: Time-Series Modelling with Multi-Scale Log-Signature Compression for State-Space Modelscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Advancing Intelligent Sequence Modeling: Evolution, Trade-offs, and Applications of State-Space Architectures from S4 to Mamba
  2. 2026A Simple State Space Model Excels at Multivariate Time Series Classification
  3. 2024Enhanced Structured State Space Models via Grouped FIR Filtering and Attention Sink Mechanisms
  4. 2022Simplifying and Understanding State Space Models with Diagonal Linear RNNs
  5. 2026An explicit operator explains end-to-end computation in the modern neural networks used for sequence and language modeling
  6. 2023Universality of Linear Recurrences Followed by Non-linear Projections: Finite-Width Guarantees and Benefits of Complex Eigenvalues
  7. 2022Liquid Structural State-Space Models
  8. 2024Incorporating Exponential Smoothing into MLP: A Simple but Effective Sequence Model