# S0 Tuning: Zero-Overhead Adaptation of Hybrid Recurrent-Attention Models

arXiv: https://arxiv.org/abs/2604.01168 · HTML: https://mlsanity.com/paper/2604.01168/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 18 of its 23 references that are in the map.
- First posted on arXiv 1 Apr 2026.

## Builds on (most cited first, 18 of 18)

- [Training Verifiers to Solve Math Word Problems](https://mlsanity.com/paper/2110.14168/) (2021)
- [Qwen3 Technical Report](https://mlsanity.com/paper/2505.09388/) (2025)
- [Evaluating Large Language Models Trained on Code](https://mlsanity.com/paper/2107.03374/) (2021)
- [Qwen2.5 Technical Report](https://mlsanity.com/paper/2412.15115/) (2024)
- [Mamba: Linear-Time Sequence Modeling with Selective State Spaces](https://mlsanity.com/paper/2312.00752/) (2023)
- [Prefix-Tuning: Optimizing Continuous Prompts for Generation](https://mlsanity.com/paper/2101.00190/) (2021)
- [Transformers are SSMs: Generalized Models and Efficient Algorithms Through Structured State Space Duality](https://mlsanity.com/paper/2405.21060/) (2024)
- [Spider: A Large-Scale Human-Labeled Dataset for Complex and Cross-Domain Semantic Parsing and Text-to-SQL Task](https://mlsanity.com/paper/1809.08887/) (2018)
- [Gated Delta Networks: Improving Mamba2 with Delta Rule](https://mlsanity.com/paper/2412.06464/) (2024)
- [Mamba-3: Improved Sequence Modeling using State Space Principles](https://mlsanity.com/paper/2603.15569/) (2026)
- [Falcon-H1: A Family of Hybrid-Head Language Models Redefining Efficiency and Performance](https://mlsanity.com/paper/2507.22448/) (2025)
- [A Systematic Analysis of Hybrid Linear Attention](https://mlsanity.com/paper/2507.06457/) (2025)
- [Parameter-Efficient Fine-Tuning of State Space Models](https://mlsanity.com/paper/2410.09016/) (2024)
- [MambaPEFT: Exploring Parameter-Efficient Fine-Tuning for Mamba](https://mlsanity.com/paper/2411.03855/) (2024)
- [SSMLoRA: Enhancing Low-Rank Adaptation with State Space Model](https://mlsanity.com/paper/2502.04958/) (2025)
- [Parameter Efficient Mamba Tuning via Projector-targeted Diagonal-centric Linear Transformation](https://mlsanity.com/paper/2411.15224/) (2024)
- [State Soup: In-Context Skill Learning, Retrieval and Mixing](https://mlsanity.com/paper/2406.08423/) (2024)
- [Memba: Membrane-driven Parameter-Efficient Fine-Tuning for Mamba](https://mlsanity.com/paper/2506.18184/) (2025)

## Built on by (most cited first, 2 of 2)

- [Where Should LoRA Go? Component-Type Placement in Hybrid Language Models](https://mlsanity.com/paper/2604.22127/) (2026)
- [LatentPort: Beyond KV Cache - Cross-Model Transfer of Recurrent Memory in Hybrid Language Models: A 4B-to-9B Hybrid-State Handoff Without Target Prefix Replay](https://mlsanity.com/paper/2609.25053/) (2026)

## Abstract (arXiv)

Using roughly 48 execution-verified HumanEval training solutions, tuning a single initial state matrix per recurrent layer, with zero inference overhead, outperforms LoRA by +10.8 pp (p < 0.001) on HumanEval. The method, which we call S0 tuning, optimizes one state matrix per recurrent layer while freezing all model weights. On Qwen3.5-4B (GatedDeltaNet hybrid), S0 tuning improves greedy pass@1 by +23.6 +/- 1.7 pp (10 seeds). On FalconH1-7B (Mamba-2 hybrid), S0 reaches 71.8% +/- 1.3 and LoRA reaches 71.4% +/- 2.4 (3 seeds), statistically indistinguishable at this sample size while requiring no weight merging. Cross-domain transfer is significant on MATH-500 (+4.8 pp, p = 0.00002, 8 seeds) and GSM8K (+2.8 pp, p = 0.0003, 10 seeds); a text-to-SQL benchmark (Spider) shows no transfer, consistent with the trajectory-steering mechanism. A prefix-tuning control on a pure Transformer (Qwen2.5-3B) degrades performance by -13.9 pp under all nine configurations tested. On Qwen3.5, a per-step state-offset variant reaches +27.1 pp, above both S0 and LoRA but with per-step inference cost. Taken together, the results show that recurrent state initialization is a strong zero-inference-overhead PEFT surface for hybrid language models when verified supervision is scarce. The tuned state is a ~48 MB file; task switching requires no weight merging or model reload. Code and library: https://github.com/jackyoung27/s0-tuning.