mlsanityData to 7 Oct 2026

Graph Signal Processing Meets Mamba2: Adaptive Filter Bank via Delta Modulation

arXiv 2603.22333 ↗First posted 20 Mar 2026cs.LG cs.AIcited by 0

Not on the map. Why?

Abstract (arXiv)

State-space models (SSMs) offer efficient alternatives to attention with linear-time recurrence. Mamba2, a recent SSM-based language model, uses selective input gating and a multi-head structure, enabling parallel computation and strong benchmark performance. However, its multi-head recurrence operates independently without structured utilization or analysis. In this work, we propose a novel method called Hierarchical ADaptive filter bank for Efficient SSMs (HADES), a Graph Signal Processing (GSP)-inspired framework that reinterprets Mamba2 as an adaptive filter bank on a line graph. Our hierarchical architecture introduces two filter types: shared filters for global low-pass behavior and expert filters for local high-pass behavior, achieved through structured bias on the parameter Delta. HADES achieves comparable performance to baseline models including Mamba2 across various benchmarks in language modeling, commonsense reasoning, and long-context retrieval, while using only 58.9% of the original parameters. In this regard, HADES bridges GSP and neural sequence modeling, enabling efficient, hierarchical, and interpretable filtering within state-space models.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 39 of its 46 references that are in the map. First posted on arXiv 20 Mar 2026.

Builds on

39 of 46 references.

References · 39

  1. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,710
  2. 2023Mamba: Linear-Time Sequence Modeling with Selective State Spacescited by 4,616
  3. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,018
  4. 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layercited by 3,413
  5. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,014
  6. 2016Pointer Sentinel Mixture Modelscited by 2,992
  7. 2019PIQA: Reasoning about Physical Commonsense in Natural Languagecited by 2,818
  8. 2021Efficiently Modeling Long Sequences with Structured State Spacescited by 2,424
  9. 2020The Pile: An 800GB Dataset of Diverse Text for Language Modelingcited by 2,346
  10. 2019BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questionscited by 2,140
  11. 2018Can a Suit of Armor Conduct Electricity? A New Dataset for Open Book Question Answeringcited by 2,114
  12. 2020Transformers are RNNs: Fast Autoregressive Transformers with Linear Attentioncited by 1,979
27 more
  1. 2019Similarity of Neural Network Representations Revisitedcited by 1,695
  2. 2019Root Mean Square Layer Normalizationcited by 1,425
  3. 2024Transformers are SSMs: Generalized Models and Efficient Algorithms Through Structured State Space Dualitycited by 1,283
  4. 2024The FineWeb Datasets: Decanting the Web for the Finest Text Data at Scalecited by 1,104
  5. 2016The LAMBADA dataset: Word prediction requiring a broad discourse contextcited by 960
  6. 2024DeepSeekMoE: Towards Ultimate Expert Specialization in Mixture-of-Experts Language Modelscited by 834
  7. 2023RWKV: Reinventing RNNs for the Transformer Eracited by 814
  8. 2022Simplified State Space Layers for Sequence Modelingcited by 776
  9. 2023Retentive Network: A Successor to Transformer for Large Language Modelscited by 594
  10. 2023Gated Linear Attention Transformers with Hardware-Efficient Trainingcited by 490
  11. 2024Gated Delta Networks: Improving Mamba2 with Delta Rulecited by 483
  12. 2023Hyena Hierarchy: Towards Larger Convolutional Language Modelscited by 388
  13. 2024Parallelizing Linear Transformers with the Delta Rule over Sequence Lengthcited by 373
  14. 2023LongLoRA: Efficient Fine-tuning of Long-Context Large Language Modelscited by 243
  15. 2022Anti-Oversmoothing in Deep Vision Transformers via the Fourier Domain Analysis: From Theory to Practicecited by 136
  16. 2022Mixture of Attention Heads: Selecting Attention Heads Per Tokencited by 58
  17. 2024DeciMamba: Exploring the Length Extrapolation Potential of Mambacited by 45
  18. 2025MoM: Linear Sequence Modeling with Mixture-of-Memoriescited by 44
  19. 2021AdaGNN: Graph Neural Networks with Adaptive Frequency Response Filtercited by 43
  20. 2023ContraNorm: A Contrastive Learning Perspective on Oversmoothing and Beyondcited by 39
  21. 2025LongMamba: Enhancing Mamba's Long Context Capabilities via Training-Free Receptive Field Enlargementcited by 36
  22. 2024Understanding the differences in Foundation Models: Attention, State Space Models, and Recurrent Neural Networkscited by 26
  23. 2024Understanding and Mitigating Bottlenecks of State Space Models through the Lens of Recency and Over-smoothingcited by 17
  24. 2021Beyond Low-pass Filtering: Graph Convolutional Networks with Automatic Filteringcited by 14
  25. 2023Focus Your Attention (with Adaptive IIR Filters)cited by 12
  26. 2023Graph Convolutions Enrich the Self-Attention in Transformers!cited by 11
  27. 2021Node-Variant Graph Filters in Graph Neural Networkscited by 7

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2024Graph-Mamba: Towards Long-Range Graph Sequence Modeling with Selective State Spaces
  2. 2025Message-Passing State-Space Models: Improving Graph Learning with Modern Sequence Modeling
  3. 2024An Empirical Study of Mamba-based Language Models
  4. 2024Enhanced Structured State Space Models via Grouped FIR Filtering and Attention Sink Mechanisms
  5. 2025On the Expressivity of Selective State-Space Layers: A Multivariate Polynomial Approach
  6. 2026InfoMamba: An Attention-Free Hybrid Mamba-Transformer Model
  7. 2024The Hidden Attention of Mamba Models
  8. 2025Achilles' Heel of Mamba: Essential difficulties of the Mamba architecture demonstrated by synthetic data