Graph Signal Processing Meets Mamba2: Adaptive Filter Bank via Delta Modulation
Not on the map. Why?
Abstract (arXiv)
State-space models (SSMs) offer efficient alternatives to attention with linear-time recurrence. Mamba2, a recent SSM-based language model, uses selective input gating and a multi-head structure, enabling parallel computation and strong benchmark performance. However, its multi-head recurrence operates independently without structured utilization or analysis. In this work, we propose a novel method called Hierarchical ADaptive filter bank for Efficient SSMs (HADES), a Graph Signal Processing (GSP)-inspired framework that reinterprets Mamba2 as an adaptive filter bank on a line graph. Our hierarchical architecture introduces two filter types: shared filters for global low-pass behavior and expert filters for local high-pass behavior, achieved through structured bias on the parameter Delta. HADES achieves comparable performance to baseline models including Mamba2 across various benchmarks in language modeling, commonsense reasoning, and long-context retrieval, while using only 58.9% of the original parameters. In this regard, HADES bridges GSP and neural sequence modeling, enabling efficient, hierarchical, and interpretable filtering within state-space models.
Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 39 of its 46 references that are in the map. First posted on arXiv 20 Mar 2026.
Builds on
39 of 46 references.
References · 39
- 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,710
- 2023Mamba: Linear-Time Sequence Modeling with Selective State Spacescited by 4,616
- 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,018
- 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layercited by 3,413
- 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,014
- 2016Pointer Sentinel Mixture Modelscited by 2,992
- 2019PIQA: Reasoning about Physical Commonsense in Natural Languagecited by 2,818
- 2021Efficiently Modeling Long Sequences with Structured State Spacescited by 2,424
- 2020The Pile: An 800GB Dataset of Diverse Text for Language Modelingcited by 2,346
- 2019BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questionscited by 2,140
- 2018Can a Suit of Armor Conduct Electricity? A New Dataset for Open Book Question Answeringcited by 2,114
- 2020Transformers are RNNs: Fast Autoregressive Transformers with Linear Attentioncited by 1,979
27 more
- 2019Similarity of Neural Network Representations Revisitedcited by 1,695
- 2019Root Mean Square Layer Normalizationcited by 1,425
- 2024Transformers are SSMs: Generalized Models and Efficient Algorithms Through Structured State Space Dualitycited by 1,283
- 2024The FineWeb Datasets: Decanting the Web for the Finest Text Data at Scalecited by 1,104
- 2016The LAMBADA dataset: Word prediction requiring a broad discourse contextcited by 960
- 2024DeepSeekMoE: Towards Ultimate Expert Specialization in Mixture-of-Experts Language Modelscited by 834
- 2023RWKV: Reinventing RNNs for the Transformer Eracited by 814
- 2022Simplified State Space Layers for Sequence Modelingcited by 776
- 2023Retentive Network: A Successor to Transformer for Large Language Modelscited by 594
- 2023Gated Linear Attention Transformers with Hardware-Efficient Trainingcited by 490
- 2024Gated Delta Networks: Improving Mamba2 with Delta Rulecited by 483
- 2023Hyena Hierarchy: Towards Larger Convolutional Language Modelscited by 388
- 2024Parallelizing Linear Transformers with the Delta Rule over Sequence Lengthcited by 373
- 2023LongLoRA: Efficient Fine-tuning of Long-Context Large Language Modelscited by 243
- 2022Anti-Oversmoothing in Deep Vision Transformers via the Fourier Domain Analysis: From Theory to Practicecited by 136
- 2022Mixture of Attention Heads: Selecting Attention Heads Per Tokencited by 58
- 2024DeciMamba: Exploring the Length Extrapolation Potential of Mambacited by 45
- 2025MoM: Linear Sequence Modeling with Mixture-of-Memoriescited by 44
- 2021AdaGNN: Graph Neural Networks with Adaptive Frequency Response Filtercited by 43
- 2023ContraNorm: A Contrastive Learning Perspective on Oversmoothing and Beyondcited by 39
- 2025LongMamba: Enhancing Mamba's Long Context Capabilities via Training-Free Receptive Field Enlargementcited by 36
- 2024Understanding the differences in Foundation Models: Attention, State Space Models, and Recurrent Neural Networkscited by 26
- 2024Understanding and Mitigating Bottlenecks of State Space Models through the Lens of Recency and Over-smoothingcited by 17
- 2021Beyond Low-pass Filtering: Graph Convolutional Networks with Automatic Filteringcited by 14
- 2023Focus Your Attention (with Adaptive IIR Filters)cited by 12
- 2023Graph Convolutions Enrich the Self-Attention in Transformers!cited by 11
- 2021Node-Variant Graph Filters in Graph Neural Networkscited by 7
Built on by
Not cited yet.
Similar, not cited
Close in wording, no citation either way
- 2024Graph-Mamba: Towards Long-Range Graph Sequence Modeling with Selective State Spaces
- 2025Message-Passing State-Space Models: Improving Graph Learning with Modern Sequence Modeling
- 2024An Empirical Study of Mamba-based Language Models
- 2024Enhanced Structured State Space Models via Grouped FIR Filtering and Attention Sink Mechanisms
- 2025On the Expressivity of Selective State-Space Layers: A Multivariate Polynomial Approach
- 2026InfoMamba: An Attention-Free Hybrid Mamba-Transformer Model
- 2024The Hidden Attention of Mamba Models
- 2025Achilles' Heel of Mamba: Essential difficulties of the Mamba architecture demonstrated by synthetic data