mlsanityData to 14 Oct 2026

Disposition Distillation at Small Scale: A Three-Arc Negative Result

arXiv 2604.11867 ↗First posted 13 Apr 2026cs.LG cs.AIcited by 0

Not on the map. Why?

Abstract (arXiv)

We set out to train behavioral dispositions (self-verification, uncertainty acknowledgment, feedback integration) into small language models (0.6B to 2.3B effective parameters) through a four-stage all-MIT distillation pipeline, with follow-on experiments on inference-time attention-head interventions and a frozen-base confidence-gated sidecar. An internal draft reported +33.9-point MCAS and +15.3-point HumanEval gains on a Qwen3-0.6B student; a second-pass sanity check falsified both numbers before publication. The HumanEval delta was a truncation artifact (n_predict=512) that inverted to -8.0 points at n_predict=1024; the MCAS gain disappeared under apples-to-apples scoring. That falsification triggered three subsequent arcs. Across (1) SFT/DPO LoRA on three model families and two domains, (2) inference-time attention-head tempering on o_proj, and (3) a training-free frozen-base sidecar reading the final-token hidden state h_last, we find no operator that moves judge-measured disposition without damaging content or collapsing into stylistic mimicry. The failure is consistent across five models (Qwen3-0.6B, Qwen3-1.7B, Qwen3.5-0.8B, Gemma 4 E2B, and SmolLM2-1.7B-Instruct). A within-distribution cross-validation pass (AUC=0.683) collapsed to chance on fresh prompts (AUC=0.516). We contribute a three-arc negative result with mechanism, a two-failure-mode taxonomy for linear h_last probes, and an honest falsification pipeline that converts the class of false positives we ourselves produced into publishable negatives. As an independent finding, Gemma 4 E2B exhibits near-complete confidence-correctness decoupling on the Chef domain (assertion asymmetry -0.009; the model asserts at 91% regardless of correctness).

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 15 of its 26 references that are in the map. First posted on arXiv 13 Apr 2026.

Builds on

15 of 26 references.

References · 15

  1. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
  2. 2015Distilling the Knowledge in a Neural Networkcited by 10,324
  3. 2021Evaluating Large Language Models Trained on Codecited by 8,320
  4. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 8,065
  5. 2020Transformer Feed-Forward Layers Are Key-Value Memoriescited by 1,336
  6. 2023Representation Engineering: A Top-Down Approach to AI Transparencycited by 1,336
  7. 2025Kimi k1.5: Scaling Reinforcement Learning with LLMscited by 1,210
  8. 2024ChatGLM: A Family of Large Language Models from GLM-130B to GLM-4 All Toolscited by 1,121
  9. 2023Steering Llama 2 via Contrastive Activation Additioncited by 1,085
  10. 2022STaR: Bootstrapping Reasoning With Reasoningcited by 940
  11. 2021Probing Classifiers: Promises, Shortcomings, and Advancescited by 876
  12. 2023Distilling Step-by-Step! Outperforming Larger Language Models with Less Training Data and Smaller Model Sizescited by 737
3 more
  1. 2019Designing and Interpreting Probes with Control Taskscited by 640
  2. 2025MiniMax-01: Scaling Foundation Models with Lightning Attentioncited by 166
  3. 2024APOLLO: SGD-like Memory, AdamW-level Performancecited by 54

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026The Asymmetric Effects of Knowledge Distillation on Bias in Small Language Models
  2. 2026When Decodability Is Not Enough: Logical Validity Representations, Behavioral Dissociation, and Causal Tests in Language Models
  3. 2026Introspection Fine-Tuning (IFT): Training Small LLMs to Introspect
  4. 2026Reinforcement Inference: Leveraging Uncertainty for Self-Correcting Language Model Reasoning
  5. 2026Post-Training Leaves Behavioral Shadows on Unrelated Decisions
  6. 2026A Systematic Study of Small Language Models on Abstract Reasoning Tasks
  7. 2026Backtracking When It Strays: Mitigating Dual Exposure Biases in LLM Reasoning Distillation
  8. 2026Disentangling Self-Distillation: Measuring and Modeling Acquisition and Retention