mlsanityData to 7 Oct 2026

Hydra: Bidirectional State Space Models Through Generalized Matrix Mixers

arXiv 2407.09941 ↗First posted 13 Jul 2024cs.LG cs.AIcited by 43

On the map: State-space sequence models (S4, Mamba) → Core SSM architectures for language

Abstract (arXiv)

A wide array of sequence models are built on a framework modeled after Transformers, comprising alternating sequence mixer and channel mixer layers. This paper studies a unifying matrix mixer view of sequence mixers that can be conceptualized as a linear map on the input sequence. This framework encompasses a broad range of well-known sequence models, including the self-attention of Transformers as well as recent strong alternatives such as structured state space models (SSMs), and allows understanding downstream characteristics such as efficiency and expressivity through properties of their structured matrix class. We identify a key axis of matrix parameterizations termed sequence alignment, which increases the flexibility and performance of matrix mixers, providing insights into the strong performance of Transformers and recent SSMs such as Mamba. Furthermore, the matrix mixer framework offers a systematic approach to developing sequence mixers with desired properties, allowing us to develop several new sub-quadratic sequence models. In particular, we propose a natural bidirectional extension of the Mamba model (Hydra), parameterized as a quasiseparable matrix mixer, which demonstrates superior performance over other sequence models including Transformers on non-causal tasks. As a drop-in replacement for attention layers, Hydra outperforms BERT by 0.8 points on the GLUE benchmark and ViT by 2% Top-1 accuracy on ImageNet.

Key facts. Cited by 43 papers in the map (data through 7 Oct 2026). Placed in Core SSM architectures for language, one of 10 directions in State-space sequence models (S4, Mamba). Builds on 41 of its 49 references that are in the map. First posted on arXiv 13 Jul 2024.

Builds on

41 of 49 references.

Closest ancestors · 5

Same direction, newest first

  1. 2024Transformers are SSMs: Generalized Models and Efficient Algorithms Through Structured State Space Duality31 May
  2. 2024Simple linear attention language models balance the recall-throughput tradeoff28 Feb
  3. 2023Gated Linear Attention Transformers with Hardware-Efficient Training11 Dec
  4. 2023Mamba: Linear-Time Sequence Modeling with Selective State Spaces1 Dec
  5. 2021Efficiently Modeling Long Sequences with Structured State Spaces31 Oct

Other references · 36

  1. 2017Attention Is All You Needcited by 59,932
  2. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  3. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  4. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  5. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  6. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,082
  7. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,854
  8. 2020Training data-efficient image transformers & distillation through attentioncited by 4,560
  9. 2014Neural Machine Translation by Jointly Learning to Align and Translatecited by 4,530
  10. 2022A ConvNet for the 2020scited by 3,545
  11. 2020Transformers are RNNs: Fast Autoregressive Transformers with Linear Attentioncited by 1,979
  12. 2020Linformer: Self-Attention with Linear Complexitycited by 1,462
24 more
  1. 2021MLP-Mixer: An all-MLP Architecture for Visioncited by 1,441
  2. 2024Vision Mamba: Efficient Visual Representation Learning with Bidirectional State Space Modelcited by 1,051
  3. 2023Retentive Network: A Successor to Transformer for Large Language Modelscited by 594
  4. 2022Hungry Hungry Hippos: Towards Language Modeling with State Space Modelscited by 495
  5. 2022On the Parameterization and Initialization of Diagonal State Space Modelscited by 418
  6. 2023Resurrecting Recurrent Neural Networks for Long Sequencescited by 397
  7. 2023Hyena Hierarchy: Towards Larger Convolutional Language Modelscited by 388
  8. 2021FNet: Mixing Tokens with Fourier Transformscited by 343
  9. 2022Long Range Language Modeling via Gated State Spacescited by 265
  10. 2024Repeat After Me: Transformers are Better than State Space Models at Copyingcited by 208
  11. 2023Zoology: Measuring and Improving Recall in Efficient Language Modelscited by 188
  12. 2022It's Raw! Audio Generation with State-Space Modelscited by 171
  13. 2021CKConv: Continuous Kernel Convolution For Sequential Datacited by 117
  14. 2024Caduceus: Bi-Directional Equivariant Long-Range DNA Sequence Modelingcited by 108
  15. 2022Monarch: Expressive Structured Matrices for Efficient and Accurate Trainingcited by 100
  16. 2022The Devil in Linear Transformercited by 98
  17. 2021How to Train BERT with an Academic Budgetcited by 80
  18. 2019Learning Fast Algorithms for Linear Transforms Using Butterfly Factorizationscited by 73
  19. 2023Monarch Mixer: A Simple Sub-Quadratic GEMM-Based Architecturecited by 62
  20. 2022Pretraining Without Attentioncited by 51
  21. 2023Toeplitz Neural Network for Sequence Modelingcited by 43
  22. 2020Kaleidoscope: An Efficient, Learnable Representation For All Structured Linear Mapscited by 36
  23. 2023MosaicBERT: A Bidirectional Encoder Optimized for Fast Pretrainingcited by 33
  24. 2023Multi-Head State Space Model for Speech Recognitioncited by 14

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 7; 2025: 27; 2026: 9.

43 papers.

Sort:

By area:

  1. 2024Transformers to SSMs: Distilling Quadratic Knowledge to Subquadratic Modelscited by 66
  2. 2024A Survey of Mambacited by 39
  3. 2024Mamba Policy: Towards Efficient 3D Diffusion Policy with Hybrid Selective State Modelscited by 22
  4. 2025TempoPFN: Synthetic Pre-training of Linear RNNs for Zero-shot Time Series Forecastingcited by 16
  5. 2025Massive Memorization with Hundreds of Trillions of Parameters for Sequential Transducer Generative Recommenderscited by 9
  6. 2024Autocorrelation Matters: Understanding the Role of Initialization Schemes for State Space Modelscited by 8
  7. 2025MamBEV: Enabling State Space Models to Learn Birds-Eye-View Representationscited by 8
  8. 2025Block-Biased Mamba for Long-Range Sequence Processingcited by 7
  9. 2025Single-Pass Document Scanning for Question Answeringcited by 6
  10. 2025Unveiling Audio Deepfake Origins: A Deep Metric learning And Conformer Network Approach With Ensemble Fusioncited by 5
  11. 2025Linear Attention for Efficient Bidirectional Sequence Modelingcited by 4
  12. 2025State-Space Models for Tabular Prior-Data Fitted Networkscited by 4
31 more
  1. 2025MambaTAD: When State-Space Models Meet Long-Range Temporal Action Detectioncited by 4
  2. 2024MambaFoley: Foley Sound Generation using Selective State-Space Modelscited by 3
  3. 2024NIMBA: Towards Robust and Principled Processing of Point Clouds With SSMscited by 3
  4. 2025Holistic Surgical Phase Recognition with Hierarchical Input Dependent State Space Modelscited by 3
  5. 2025Technologies on Effectiveness and Efficiency: A Survey of State Spaces Modelscited by 2
  6. 2025Knowing When to Quit: Probabilistic Early Exits for Speech Separationcited by 2
  7. 2025Chimera: State Space Models Beyond Sequencescited by 2
  8. 2025Hankel Singular Value Regularization for Highly Compressible State Space Modelscited by 2
  9. 2026PACER: Blockwise Pre-verification for Speculative Decoding with Adaptive Lengthcited by 2
  10. 2026RoVE: Rotary Value Embeddings Attention for Relative Position-dependent Value Pathwayscited by 2
  11. 2025Developing multilingual speech synthesis system for Ojibwe, Mi'kmaq, and Maliseetcited by 1
  12. 2025LBMamba: Locally Bi-directional Mambacited by 1
  13. 2025MVHybrid: Improving Spatial Transcriptomics Prediction with Hybrid State Space-Vision Transformer Backbone in Pathology Vision Foundation Modelscited by 1
  14. 2025Resting-state fMRI Analysis using Quantum Time-series Transformercited by 1
  15. 2025On Structured State-Space Dualitycited by 1
  16. 2025HieraMamba: Video Temporal Grounding via Hierarchical Anchor-Mamba Poolingcited by 1
  17. 2025AdaPerceiver: Transformers with Adaptive Width, Depth, and Tokenscited by 1
  18. 2026XLSR-MamBo: Scaling the Hybrid Mamba-Attention Backbone for Audio Deepfake Detectioncited by 1
  19. 2026CLIMP: Contrastive Language-Image Mamba Pretrainingcited by 1
  20. 2026Tydra: An Efficient Hybrid Model for Tabular Datacited by 1
  21. 2024Exploring Applications of State Space Models and Advanced Training Techniques in Sequential Recommendations: A Comparative Study on Efficiency and Performancecited by 0
  22. 2025Is Long Range Sequential Modeling Necessary For Colorectal Tumor Segmentation?cited by 0
  23. 2025MobileViM: A Light-weight and Dimension-independent Vision Mamba for 3D Medical Image Analysiscited by 0
  24. 2025HybridTM: Combining Transformer and Mamba for 3D Semantic Segmentationcited by 0
  25. 2025JustDense: Just using Dense instead of Sequence Mixer for Time Series analysiscited by 0
  26. 2025Customizing the Inductive Biases of Softmax Attention using Structured Matricescited by 0
  27. 2025An overview of neural architectures for self-supervised audio representation learning from masked spectrogramscited by 0
  28. 2026Breaking the Bottlenecks: Scalable Diffusion Models for 3D Molecular Generationcited by 0
  29. 2026Spatial Priors via Space Filling Curves for Small and Limited Data Vision Transformerscited by 0
  30. 2026BiMamba2 Masked Discrete-Unit Prediction for Multilingual Speech Representation for Unsupervised Speech in the Wild Challengecited by 0
  31. 2026Adapting Linear-Time Architectures for Tabular In-Context Learningcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Enhanced Structured State Space Models via Grouped FIR Filtering and Attention Sink Mechanisms
  2. 2025Design Principles for Sequence Models via Coefficient Dynamics
  3. 2025Advancing Intelligent Sequence Modeling: Evolution, Trade-offs, and Applications of State-Space Architectures from S4 to Mamba
  4. 2026Multi-Mixer Models: Flexible Sequence Modeling with Shared Representations
  5. 2026Placement Is Free, Composition Is Not: The Latin Square as a Provably-Balanced Construction for Heterogeneous Sequence-Mixer Stacks
  6. 2025On the Expressivity of Selective State-Space Layers: A Multivariate Polynomial Approach
  7. 2025Achilles' Heel of Mamba: Essential difficulties of the Mamba architecture demonstrated by synthetic data
  8. 2024MambaMixer: Efficient Selective State Space Models with Dual Token and Channel Selection