S0 Tuning: Zero-Overhead Adaptation of Hybrid Recurrent-Attention Models
Not on the map. Why?
Abstract (arXiv)
Using roughly 48 execution-verified HumanEval training solutions, tuning a single initial state matrix per recurrent layer, with zero inference overhead, outperforms LoRA by +10.8 pp (p < 0.001) on HumanEval. The method, which we call S0 tuning, optimizes one state matrix per recurrent layer while freezing all model weights. On Qwen3.5-4B (GatedDeltaNet hybrid), S0 tuning improves greedy pass@1 by +23.6 +/- 1.7 pp (10 seeds). On FalconH1-7B (Mamba-2 hybrid), S0 reaches 71.8% +/- 1.3 and LoRA reaches 71.4% +/- 2.4 (3 seeds), statistically indistinguishable at this sample size while requiring no weight merging. Cross-domain transfer is significant on MATH-500 (+4.8 pp, p = 0.00002, 8 seeds) and GSM8K (+2.8 pp, p = 0.0003, 10 seeds); a text-to-SQL benchmark (Spider) shows no transfer, consistent with the trajectory-steering mechanism. A prefix-tuning control on a pure Transformer (Qwen2.5-3B) degrades performance by -13.9 pp under all nine configurations tested. On Qwen3.5, a per-step state-offset variant reaches +27.1 pp, above both S0 and LoRA but with per-step inference cost. Taken together, the results show that recurrent state initialization is a strong zero-inference-overhead PEFT surface for hybrid language models when verified supervision is scarce. The tuned state is a ~48 MB file; task switching requires no weight merging or model reload. Code and library: https://github.com/jackyoung27/s0-tuning.
Key facts. Cited by 2 papers in the map (data through 7 Oct 2026). Builds on 18 of its 23 references that are in the map. First posted on arXiv 1 Apr 2026.
Builds on
18 of 23 references.
References · 18
- 2021Training Verifiers to Solve Math Word Problemscited by 9,743
- 2025Qwen3 Technical Reportcited by 8,195
- 2021Evaluating Large Language Models Trained on Codecited by 8,053
- 2024Qwen2.5 Technical Reportcited by 4,713
- 2023Mamba: Linear-Time Sequence Modeling with Selective State Spacescited by 4,616
- 2021Prefix-Tuning: Optimizing Continuous Prompts for Generationcited by 3,878
- 2024Transformers are SSMs: Generalized Models and Efficient Algorithms Through Structured State Space Dualitycited by 1,283
- 2018Spider: A Large-Scale Human-Labeled Dataset for Complex and Cross-Domain Semantic Parsing and Text-to-SQL Taskcited by 1,120
- 2024Gated Delta Networks: Improving Mamba2 with Delta Rulecited by 483
- 2026Mamba-3: Improved Sequence Modeling using State Space Principlescited by 76
- 2025Falcon-H1: A Family of Hybrid-Head Language Models Redefining Efficiency and Performancecited by 52
- 2025A Systematic Analysis of Hybrid Linear Attentioncited by 38
6 more
- 2024Parameter-Efficient Fine-Tuning of State Space Modelscited by 11
- 2024MambaPEFT: Exploring Parameter-Efficient Fine-Tuning for Mambacited by 11
- 2025SSMLoRA: Enhancing Low-Rank Adaptation with State Space Modelcited by 5
- 2024Parameter Efficient Mamba Tuning via Projector-targeted Diagonal-centric Linear Transformationcited by 4
- 2024State Soup: In-Context Skill Learning, Retrieval and Mixingcited by 3
- 2025Memba: Membrane-driven Parameter-Efficient Fine-Tuning for Mambacited by 2
Built on by
Citing papers per year: 2026: 2.
2 papers.
Similar, not cited
Close in wording, no citation either way
- 2024NoRA: Nested Low-Rank Adaptation for Efficient Fine-Tuning Large Models
- 2026Learning Rate Matters: Vanilla LoRA May Suffice for LLM Fine-tuning
- 2026One Rate Is Not Enough: Adaptive Anisotropic Learning Rates for LoRA Fine-Tuning
- 2024DoRA: Weight-Decomposed Low-Rank Adaptation
- 2025ScaLoRA: Optimally Scaled Low-Rank Adaptation for Efficient High-Rank Fine-Tuning
- 2026Parameter-Efficient Fine-Tuning with Learnable Rank
- 2026Beyond Low-Rank Parameterization: Narrowing the Gap Between LoRA and Full Fine-Tuning via Gradient Decomposition
- 2026SSM Adapters via Hankel Reduced-order Modeling: Injection Site Determines Task Suitability in Long-Context Fine-Tuning