mlsanityData to 7 Oct 2026

Parameter-Efficient Fine-Tuning of State Space Models

arXiv 2410.09016 ↗First posted 11 Oct 2024cs.LG cs.CLcited by 11

On the map: State-space sequence models (S4, Mamba) → Core SSM architectures for language

Abstract (arXiv)

Deep State Space Models (SSMs), such as Mamba (Gu & Dao, 2024), have become powerful tools for language modeling, offering high performance and linear scalability with sequence length. However, the application of parameter-efficient fine-tuning (PEFT) methods to SSM-based models remains largely underexplored. We start by investigating two fundamental questions on existing PEFT methods: (i) How do they perform on SSM-based models? (ii) Which parameters should they target for optimal results? Our analysis shows that LoRA and its variants consistently outperform all other PEFT methods. While LoRA is effective for linear projection matrices, it fails on SSM modules-yet still outperforms other methods applicable to SSMs, indicating their limitations. This underscores the need for a specialized SSM tuning approach. To address this, we propose Sparse Dimension Tuning (SDT), a PEFT method tailored for SSM modules. Combining SDT for SSMs with LoRA for linear projection matrices, we achieve state-of-the-art performance across extensive experiments.

Key facts. Cited by 11 papers in the map (data through 7 Oct 2026). Placed in Core SSM architectures for language, one of 10 directions in State-space sequence models (S4, Mamba). Builds on 45 of its 52 references that are in the map. First posted on arXiv 11 Oct 2024.

Builds on

45 of 52 references.

Closest ancestors · 6

Same direction, newest first

  1. 2024Transformers are SSMs: Generalized Models and Efficient Algorithms Through Structured State Space Duality31 May
  2. 2024Jamba: A Hybrid Transformer-Mamba Language Model28 Mar
  3. 2024Can Mamba Learn How to Learn? A Comparative Study on In-Context Learning Tasks6 Feb
  4. 2023Mamba: Linear-Time Sequence Modeling with Selective State Spaces1 Dec
  5. 2022Diagonal State Spaces are as Effective as Structured State Spaces27 Mar
  6. 2021Efficiently Modeling Long Sequences with Structured State Spaces31 Oct

Other references · 39

  1. 2017Attention Is All You Needcited by 59,932
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2023GPT-4 Technical Reportcited by 18,132
  4. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,854
  5. 2019Parameter-Efficient Transfer Learning for NLPcited by 4,195
  6. 2021Prefix-Tuning: Optimizing Continuous Prompts for Generationcited by 3,878
  7. 2021The Power of Scale for Parameter-Efficient Prompt Tuningcited by 3,769
  8. 2014Deep Learning Face Attributes in the Wildcited by 3,679
  9. 2020The Pile: An 800GB Dataset of Diverse Text for Language Modelingcited by 2,346
  10. 2020Transformers are RNNs: Fast Autoregressive Transformers with Linear Attentioncited by 1,979
  11. 2021BitFit: Simple Parameter-efficient Fine-tuning for Transformer-based Masked Language-modelscited by 1,234
  12. 2014How transferable are features in deep neural networks?cited by 1,216
27 more
  1. 2018Spider: A Large-Scale Human-Labeled Dataset for Complex and Cross-Domain Semantic Parsing and Text-to-SQL Taskcited by 1,120
  2. 2021Towards a Unified View of Parameter-Efficient Transfer Learningcited by 863
  3. 2023RWKV: Reinventing RNNs for the Transformer Eracited by 814
  4. 2024DoRA: Weight-Decomposed Low-Rank Adaptationcited by 740
  5. 2021GPT Understands, Toocited by 715
  6. 2021Combining Recurrent, Convolutional, and Continuous-time Models with Linear State-Space Layerscited by 607
  7. 2020HiPPO: Recurrent Memory with Optimal Polynomial Projectionscited by 600
  8. 2023Retentive Network: A Successor to Transformer for Large Language Modelscited by 594
  9. 2019SAMSum Corpus: A Human-annotated Dialogue Dataset for Abstractive Summarizationcited by 530
  10. 2022Hungry Hungry Hippos: Towards Language Modeling with State Space Modelscited by 495
  11. 2013DeCAF: A Deep Convolutional Activation Feature for Generic Visual Recognitioncited by 422
  12. 2022On the Parameterization and Initialization of Diagonal State Space Modelscited by 418
  13. 2021PICARD: Parsing Incrementally for Constrained Auto-Regressive Decoding from Language Modelscited by 402
  14. 2024LoRA+: Efficient Low Rank Adaptation of Large Modelscited by 365
  15. 2023LLM-Adapters: An Adapter Family for Parameter-Efficient Fine-Tuning of Large Language Modelscited by 317
  16. 2022LIFT: Language-Interfaced Fine-Tuning for Non-Language Machine Learning Taskscited by 146
  17. 2020DART: Open-Domain Structured Data Record to Text Generationcited by 144
  18. 2023Task-Specific Skill Localization in Fine-tuned Language Modelscited by 104
  19. 2023The Expressive Power of Low-Rank Adaptationcited by 100
  20. 2021Pufferfish: Communication-efficient Models At No Extra Costcited by 61
  21. 2023On the Role of Attention in Prompt-tuningcited by 49
  22. 2024LoRA Training in the NTK Regime has No Spurious Local Minimacited by 48
  23. 2023When Do Prompting and Prefix-Tuning Work? A Theory of Capabilities and Limitationscited by 41
  24. 2023Universality and Limitations of Prompt Tuningcited by 30
  25. 2023Cuttlefish: Low-Rank Model Training without All the Tuningcited by 25
  26. 2023Sparse is Enough in Fine-tuning Pre-trained Large Language Modelscited by 21
  27. 2023The Expressive Power of Tuning Only the Normalization Layerscited by 13

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 2; 2025: 3; 2026: 6.

11 papers.

  1. 2024MambaPEFT: Exploring Parameter-Efficient Fine-Tuning for Mambacited by 11
  2. 2024Expansion Span: Combining Fading Memory and Retrieval in Hybrid State Space Modelscited by 11
  3. 2025Augmenting Perceptual Super-Resolution via Image Quality Predictorscited by 7
  4. 2026SF-Mamba: Rethinking State Space Model for Visioncited by 4
  5. 2025EMRRG: Efficient Fine-Tuning Pre-trained X-ray Mamba Networks for Radiology Report Generationcited by 3
  6. 2026S0 Tuning: Zero-Overhead Adaptation of Hybrid Recurrent-Attention Modelscited by 2
  7. 2026WriteSAE: Sparse Autoencoders for Recurrent Statecited by 1
  8. 2025SAM: A Mamba-2 State-Space Audio-Language Modelcited by 0
  9. 2026Where Should LoRA Go? Component-Type Placement in Hybrid Language Modelscited by 0
  10. 2026A Reference-Free Framework for Evaluating Single-Frame ISP Pipelinescited by 0
  11. 2026Energy- and Memory-Efficient PEFT Methods for Personalized On-Device SLMs on Consumer GPUscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025SSMLoRA: Enhancing Low-Rank Adaptation with State Space Model
  2. 2023Towards Better Parameter-Efficient Fine-Tuning for Large Language Models: A Position Paper
  3. 2024State Space Models as Foundation Models: A Control Theoretic Overview
  4. 2025Uni-LoRA: One Vector is All You Need
  5. 2025State-offset Tuning: State-based Parameter-Efficient Fine-Tuning for State Space Models
  6. 2025PEFT A2Z: Parameter-Efficient Fine-Tuning Survey for Large Language and Vision Models
  7. 2025ScaLoRA: Optimally Scaled Low-Rank Adaptation for Efficient High-Rank Fine-Tuning
  8. 2026ID-LoRA: Efficient Low-Rank Adaptation Inspired by Matrix Interpolative Decomposition