mlsanityData to 14 Oct 2026

SFT Leaves a Reproducible Signature in Activation Geometry: Calibrating Post-Training Interpretability Against Run-to-Run Noise

arXiv 2610.17093 ↗First posted 12 Oct 2026cs.AIcited by 0

Not on the map. Why?

Abstract (arXiv)

A model diff credits post-training with whatever separates a base checkpoint from one post-trained descendant, presuming a rerun would land close by. On the sparse co-activation graph of Qwen3-1.7B it does not: two LoRA-SFT reruns sit over half as far apart as either sits from the base. We score each base-to-post contrast against the run-to-run band of its own recipe, and show that k runs certify no better than p=1/(k+1) under run exchangeability, however large the effect, so the band z-score is a descriptive scale. SFT leaves a large, reproducible signature: base → SFT displaces the graph by 17.8 pairwise-distance SDs beyond the mean distance between four independent LoRA-SFT runs, and the pre-registered estimator replicates on Llama-3.1-8B with Llama-Scope. At 29 steps each, four SFT and four DPO runs occupy disjoint displacement ranges, a gap 5.0× and 5.6× the spreads of the two arms, without any band. All bands use graphs built on one GPU architecture. Three other readings stop short of a verdict: DPO band counts (3/4, 1/4, 3/4 contrasts outside at 12, 29, 60 steps) show no consistent dose ordering while its displacement stays flat; light GRPO is a secondary non-detection, not an absence, and we run no equivalence test; and the graph concentration read as an RL signature is already in the zero-RL base. A pre-registered causal sweep at three edge budgets on 4,816 held-out prompts found the displaced sources load-bearing (12× a matched random control in mean, 3.5× in median at 10,000 edges), but their stage-specificity interaction reversed at two budgets and the third failed its faithfulness gate, so we claim calibrated structural displacement, not a changed mechanism. Scope: one backbone, one SFT recipe and general-text probe states for the full protocol.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 71 of its 83 references that are in the map. First posted on arXiv 12 Oct 2026.

Builds on

71 of 83 references.

References · 71

  1. 2017Proximal Policy Optimization Algorithmscited by 15,215
  2. 2024The Llama 3 Herd of Modelscited by 14,715
  3. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  4. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 9,379
  5. 2025Qwen3 Technical Reportcited by 8,820
  6. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 8,065
  7. 2020The Pile: An 800GB Dataset of Diverse Text for Language Modelingcited by 2,411
  8. 2024Gemma 2: Improving Open Language Models at a Practical Sizecited by 1,998
  9. 2023Pythia: A Suite for Analyzing Large Language Models Across Training and Scalingcited by 1,786
  10. 2025Understanding R1-Zero-Like Training: A Critical Perspectivecited by 1,345
  11. 2024Tulu 3: Pushing Frontiers in Open Language Model Post-Trainingcited by 1,053
  12. 2025Does Reinforcement Learning Really Incentivize Reasoning Capacity in LLMs Beyond the Base Model?cited by 982
38 more
  1. 2024Scaling and evaluating sparse autoencoderscited by 553
  2. 2025The Entropy Mechanism of Reinforcement Learning for Reasoning Language Modelscited by 485
  3. 2024Sparse Feature Circuits: Discovering and Editing Interpretable Causal Graphs in Language Modelscited by 435
  4. 2023Understanding the Effects of RLHF on LLM Generalisation and Diversitycited by 396
  5. 2024Gemma Scope: Open Sparse Autoencoders Everywhere All At Once on Gemma 2cited by 394
  6. 2020Fine-Tuning Pretrained Language Models: Weight Initializations, Data Orders, and Early Stoppingcited by 384
  7. 2025Persona Vectors: Monitoring and Controlling Character Traits in Language Modelscited by 345
  8. 20242 OLMo 2 Furiouscited by 340
  9. 2024Jumping Ahead: Improving Reconstruction Fidelity with JumpReLU Sparse Autoencoderscited by 300
  10. 2025Reinforcement Learning for Reasoning in Large Language Models with One Training Examplecited by 295
  11. 2023The Unlocking Spell on Base LLMs: Rethinking Alignment via In-Context Learningcited by 257
  12. 2015Convergent Learning: Do different neural networks learn the same representations?cited by 220
  13. 2024How to use and interpret activation patchingcited by 214
  14. 2024Transcoders Find Interpretable LLM Feature Circuitscited by 200
  15. 2026Scaling Monosemanticity: Extracting Interpretable Features from Claude 3 Sonnetcited by 191
  16. 2024A Mechanistic Understanding of Alignment Algorithms: A Case Study on DPO and Toxicitycited by 185
  17. 2024Fine-Tuning Enhances Existing Mechanisms: A Case Study on Entity Trackingcited by 149
  18. 2024Llama Scope: Extracting Millions of Features from Llama-3.1-8B with Sparse Autoencoderscited by 143
  19. 2023Mechanistically analyzing the effects of fine-tuning on procedurally defined taskscited by 115
  20. 2024LoRA vs Full Fine-tuning: An Illusion of Equivalencecited by 109
  21. 2025Sparse Autoencoders Trained on the Same Data Learn Different Featurescited by 93
  22. 2021The MultiBERTs: BERT Reproductions for Robustness Analysiscited by 92
  23. 2025When Thinking Fails: The Pitfalls of Reasoning for Instruction-Following in LLMscited by 53
  24. 2025Overcoming Sparsity Artifacts in Crosscoders to Interpret Chat-Tuningcited by 34
  25. 2025Narrow Finetuning Leaves Clearly Readable Traces in Activation Differencescited by 28
  26. 2025Base Models Know How to Reason, Thinking Models Learn Whencited by 27
  27. 2026Qwen-Scope: Turning Sparse Features into Development Tools for Large Language Modelscited by 21
  28. 2026Reproducing, Analyzing, and Detecting Reward Hacking in Rubric-Based Reinforcement Learningcited by 16
  29. 2026Spurious Rewards Paradox: Mechanistically Understanding How RLVR Activates Memorization Shortcuts in LLMscited by 14
  30. 2026Sanity Checks for Sparse Autoencoders: Do SAEs Beat Random Baselines?cited by 14
  31. 2025Activation Control for Efficiently Eliciting Long Chain-of-thought Ability of Language Modelscited by 13
  32. 2026On the Direction of RLVR Updates for LLM Reasoning: Identification and Exploitationcited by 13
  33. 2025The Dead Salmons of AI Interpretabilitycited by 12
  34. 2025Reinforcement Learning Fine-Tuning Enhances Activation Intensity and Diversity in the Internal Circuitry of LLMscited by 9
  35. 2026Tracing Persona Vectors Through LLM Pretrainingcited by 7
  36. 2026The Value Axis: Language Models Encode Whether They're on the Right Trackcited by 7
  37. 2026From Simulation to Enaction: Post-trained language models recognize and react to their own generationscited by 6
  38. 2025Logit Arithmetic Elicits Long Reasoning Capabilities Without Trainingcited by 4

50 most cited of 71 All 71 (JSON)

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026Procedural-skill SFT across capacity tiers: A W-Shaped pre-SFT Trajectory and Regime-Asymmetric Mechanism on 0.8B-4B Qwen3.5 Models
  2. 2026SteerCheck: Attribution Specificity and Alignment Leakage in Activation-Steering Audits
  3. 2026Similar Models Learn Differently: Final-Window Pretraining Shapes Post-Training Beyond SFT
  4. 2026Recoverable but Not Stationary:Local Linear Structures in Weights and Activations
  5. 2026What Will Post-Training Fix? Per-Problem Gains Are Shared Across Independent RL Runs, and Existing Checkpoints Predict Them Better Than A Priori Signals
  6. 2025Induction Signatures Are Not Enough: A Matched-Compute Study of Load-Bearing Structure in In-Context Learning
  7. 2026OUI as a Structural Observable: Towards an Activation-Centric View of Neural Network Training
  8. 2026Selection-Safe Conformal Early Exit: Distribution-Free Coverage under Adaptive Depth