mlsanityData to 7 Oct 2026

Procedural-skill SFT across capacity tiers: A W-Shaped pre-SFT Trajectory and Regime-Asymmetric Mechanism on 0.8B-4B Qwen3.5 Models

arXiv 2605.11907 ↗First posted 12 May 2026cs.LGcited by 0

Not on the map. Why?

Abstract (arXiv)

We measure procedural-skill SFT contribution across three Qwen3.5 dense scales (0.8B, 2B, 4B) on a 200-task / 40-skill holdout, with Claude Haiku 4.5 as a frontier reference. The corpus is 353 rows of (task + procedural-skill block, Opus chain-of-thought, judge-pass) demonstrations. Main finding. Under matched-path LLM-only scoring, the SFT-attributable procedural-Delta lift is roughly uniform across sizes: +0.070 / +0.040 / +0.075 at 0.8B / 2B / 4B. Variation in post-SFT Delta (-0.005, +0.100, +0.065) is dominated by a W-shaped pre-SFT base trajectory (-0.075, +0.060, -0.010, Haiku-4-5 at +0.030): the 5-step procedure hurts 0.8B and 4B, helps 2B, and helps frontier Haiku modestly. SFT works hardest in absolute terms where the base struggles with the procedure -- a regime-asymmetric pattern with a falsifiable prediction at 8B/14B. Methodology. (i) A bench format-compliance artifact: 83.5% of the holdout uses a deterministic ANSWER-line extractor that under-counts free-form-prose conclusions; our LLM-only re-judge reveals it was systematically biased against the curated condition. (ii) A negative-iteration sequence at 0.8B: three well-formed recipe variants cluster post-SFT curated pass-rate within a 2 pp band, constraining the absolute-pass-rate ceiling to base capacity rather than recipe. Cross-family judge validation. GPT-5.4 via OpenRouter on all 7 configurations (2800 paired episodes) agrees on the direction of every per-student finding: Cohen's kappa ≥ 0.754, agreement ≥ 93.25%, max headline Delta shift ≤ 0.035 pp. Two earlier framings -- "format-only learning at 0.8B" and "SFT contribution shrinks at 4B" -- were path-mismatch artifacts; this paper supersedes both. Single-seed evaluation; threats itemised in the paper.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 6 of its 17 references that are in the map. First posted on arXiv 12 May 2026.

Builds on

6 of 17 references.

References · 6

  1. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  2. 2025Qwen3 Technical Reportcited by 8,195
  3. 2023QLoRA: Efficient Finetuning of Quantized LLMscited by 3,355
  4. 2024Lessons from the Trenches on Reproducible Evaluation of Language Modelscited by 172
  5. 2023A Theory for Emergence of Complex Skills in Language Modelscited by 111
  6. 2023Skill-Mix: a Flexible and Expandable Family of Evaluations for AI modelscited by 64

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026Procedural Knowledge Is Not Low-Rank: Why LoRA Fails to Internalize Multi-Step Procedures
  2. 2026Skill Availability and Presentation Granularity in Large-Language-Model Agents: A Controlled SkillsBench Study
  3. 2026SKT: Skill-Use Training at Scale via Verified Synthetic Data Generation
  4. 2026SkillGLoW: Procedural-Family Skill Consolidation for Self-Improving Agents on Long-Horizon Task Streams
  5. 2026SkillJuror: Measuring How Agent Skill Organization Changes Runtime Behavior
  6. 2025Rethinking Expert Trajectory Utilization in LLM Post-training for Mathematical Reasoning
  7. 2026Procedural Knowledge at Scale Improves Reasoning
  8. 2026SkillApt: Learning When to Activate Agent Skills from Counterfactual Evidence