Procedural-skill SFT across capacity tiers: A W-Shaped pre-SFT Trajectory and Regime-Asymmetric Mechanism on 0.8B-4B Qwen3.5 Models
Not on the map. Why?
Abstract (arXiv)
We measure procedural-skill SFT contribution across three Qwen3.5 dense scales (0.8B, 2B, 4B) on a 200-task / 40-skill holdout, with Claude Haiku 4.5 as a frontier reference. The corpus is 353 rows of (task + procedural-skill block, Opus chain-of-thought, judge-pass) demonstrations. Main finding. Under matched-path LLM-only scoring, the SFT-attributable procedural-Delta lift is roughly uniform across sizes: +0.070 / +0.040 / +0.075 at 0.8B / 2B / 4B. Variation in post-SFT Delta (-0.005, +0.100, +0.065) is dominated by a W-shaped pre-SFT base trajectory (-0.075, +0.060, -0.010, Haiku-4-5 at +0.030): the 5-step procedure hurts 0.8B and 4B, helps 2B, and helps frontier Haiku modestly. SFT works hardest in absolute terms where the base struggles with the procedure -- a regime-asymmetric pattern with a falsifiable prediction at 8B/14B. Methodology. (i) A bench format-compliance artifact: 83.5% of the holdout uses a deterministic ANSWER-line extractor that under-counts free-form-prose conclusions; our LLM-only re-judge reveals it was systematically biased against the curated condition. (ii) A negative-iteration sequence at 0.8B: three well-formed recipe variants cluster post-SFT curated pass-rate within a 2 pp band, constraining the absolute-pass-rate ceiling to base capacity rather than recipe. Cross-family judge validation. GPT-5.4 via OpenRouter on all 7 configurations (2800 paired episodes) agrees on the direction of every per-student finding: Cohen's kappa ≥ 0.754, agreement ≥ 93.25%, max headline Delta shift ≤ 0.035 pp. Two earlier framings -- "format-only learning at 0.8B" and "SFT contribution shrinks at 4B" -- were path-mismatch artifacts; this paper supersedes both. Single-seed evaluation; threats itemised in the paper.
Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 6 of its 17 references that are in the map. First posted on arXiv 12 May 2026.
Builds on
6 of 17 references.
References · 6
- 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
- 2025Qwen3 Technical Reportcited by 8,195
- 2023QLoRA: Efficient Finetuning of Quantized LLMscited by 3,355
- 2024Lessons from the Trenches on Reproducible Evaluation of Language Modelscited by 172
- 2023A Theory for Emergence of Complex Skills in Language Modelscited by 111
- 2023Skill-Mix: a Flexible and Expandable Family of Evaluations for AI modelscited by 64
Built on by
Not cited yet.
Similar, not cited
Close in wording, no citation either way
- 2026Procedural Knowledge Is Not Low-Rank: Why LoRA Fails to Internalize Multi-Step Procedures
- 2026Skill Availability and Presentation Granularity in Large-Language-Model Agents: A Controlled SkillsBench Study
- 2026SKT: Skill-Use Training at Scale via Verified Synthetic Data Generation
- 2026SkillGLoW: Procedural-Family Skill Consolidation for Self-Improving Agents on Long-Horizon Task Streams
- 2026SkillJuror: Measuring How Agent Skill Organization Changes Runtime Behavior
- 2025Rethinking Expert Trajectory Utilization in LLM Post-training for Mathematical Reasoning
- 2026Procedural Knowledge at Scale Improves Reasoning
- 2026SkillApt: Learning When to Activate Agent Skills from Counterfactual Evidence