# Model Spec Midtraining: Improving How Alignment Training Generalizes

arXiv: https://arxiv.org/abs/2605.02087 · HTML: https://mlsanity.com/paper/2605.02087/

## Key facts

- Cited by 20 papers in the map (data through 14 Oct 2026).
- Builds on 28 of its 36 references that are in the map.
- First posted on arXiv 3 May 2026.

## Builds on (most cited first, 28 of 28)

- [The Llama 3 Herd of Models](https://mlsanity.com/paper/2407.21783/) (2024)
- [Deep reinforcement learning from human preferences](https://mlsanity.com/paper/1706.03741/) (2017)
- [Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback](https://mlsanity.com/paper/2204.05862/) (2022)
- [Qwen Technical Report](https://mlsanity.com/paper/2309.16609/) (2023)
- [Constitutional AI: Harmlessness from AI Feedback](https://mlsanity.com/paper/2212.08073/) (2022)
- [LIMA: Less Is More for Alignment](https://mlsanity.com/paper/2305.11206/) (2023)
- [Tulu 3: Pushing Frontiers in Open Language Model Post-Training](https://mlsanity.com/paper/2411.15124/) (2024)
- [Alignment faking in large language models](https://mlsanity.com/paper/2412.14093/) (2024)
- [Persona Vectors: Monitoring and Controlling Character Traits in Language Models](https://mlsanity.com/paper/2507.21509/) (2025)
- [SmolLM2: When Smol Goes Big -- Data-Centric Training of a Small Language Model](https://mlsanity.com/paper/2502.02737/) (2025)
- [Deliberative Alignment: Reasoning Enables Safer Language Models](https://mlsanity.com/paper/2412.16339/) (2024)
- [Chain of Thought Monitorability: A New and Fragile Opportunity for AI Safety](https://mlsanity.com/paper/2507.11473/) (2025)
- [Pretraining Language Models with Human Preferences](https://mlsanity.com/paper/2302.08582/) (2023)
- [Agentic Misalignment: How LLMs Could Be Insider Threats](https://mlsanity.com/paper/2510.05179/) (2025)
- [Natural Emergent Misalignment from Reward Hacking in Production RL](https://mlsanity.com/paper/2511.18397/) (2025)
- [Stress Testing Deliberative Alignment for Anti-Scheming Training](https://mlsanity.com/paper/2509.15541/) (2025)
- [Auditing language models for hidden objectives](https://mlsanity.com/paper/2503.10965/) (2025)
- [Specific versus General Principles for Constitutional AI](https://mlsanity.com/paper/2310.13798/) (2023)
- [Safety Pretraining: Toward the Next Generation of Safe AI](https://mlsanity.com/paper/2504.16980/) (2025)
- [Inoculation Prompting: Eliciting traits from LLMs during training can suppress them at test-time](https://mlsanity.com/paper/2510.04340/) (2025)
- [Inoculation Prompting: Instructing LLMs to misbehave at train-time improves test-time alignment](https://mlsanity.com/paper/2510.05024/) (2025)
- [Deep Ignorance: Filtering Pretraining Data Builds Tamper-Resistant Safeguards into Open-Weight LLMs](https://mlsanity.com/paper/2508.06601/) (2025)
- [Mitigating harm in language models with conditional-likelihood filtration](https://mlsanity.com/paper/2108.07790/) (2021)
- [Steering Out-of-Distribution Generalization with Concept Ablation Fine-Tuning](https://mlsanity.com/paper/2507.16795/) (2025)
- [Uncovering Deceptive Tendencies in Language Models: A Simulated Company AI Assistant](https://mlsanity.com/paper/2405.01576/) (2024)
- [Gradient Routing: Masking Gradients to Localize Computation in Neural Networks](https://mlsanity.com/paper/2410.04332/) (2024)
- [Alignment Pretraining: AI Discourse Causes Self-Fulfilling (Mis)alignment](https://mlsanity.com/paper/2601.10160/) (2026)
- [Midtraining Bridges Pretraining and Posttraining Distributions](https://mlsanity.com/paper/2510.14865/) (2025)

## Built on by (most cited first, 20 of 20)

- [Negation Neglect: When models fail to learn negations in training](https://mlsanity.com/paper/2605.13829/) (2026)
- [Persona Cartography: Charting Language Model Personality Traits in Weight Space](https://mlsanity.com/paper/2607.07916/) (2026)
- [Constitutional Midtraining: Content Presence Drives Alignment Gains](https://mlsanity.com/paper/2607.26654/) (2026)
- [Enhancing LLM Metacognition via Cognitive Pairwise Training](https://mlsanity.com/paper/2606.00869/) (2026)
- [(Mis)generalization of Helpful-only Fine-tuning](https://mlsanity.com/paper/2606.04413/) (2026)
- [Epistemic Goggles: A Pretrained Module that Induces an Epistemic Frame via Gradient Editing](https://mlsanity.com/paper/2607.01690/) (2026)
- [Shared SFT Lessons Across Alignment, Model Organisms, and Toy Models](https://mlsanity.com/paper/2607.26173/) (2026)
- [Data Attribution of Emergent Misalignment with Persona Features](https://mlsanity.com/paper/2608.11025/) (2026)
- [Synthetic Persona Pretraining: Alignment from Token Zero](https://mlsanity.com/paper/2608.13482/) (2026)
- [A Virtuous AI is an Existential Risk](https://mlsanity.com/paper/2606.13739/) (2026)
- [Oyster-II: Reinforcement Learning for Constructive Safety Alignment in Large Language Models](https://mlsanity.com/paper/2607.02914/) (2026)
- [Similar Models Learn Differently: Final-Window Pretraining Shapes Post-Training Beyond SFT](https://mlsanity.com/paper/2607.25063/) (2026)
- [Story Imprinting: AI Assistants Absorb Traits from Human Characters They Resemble](https://mlsanity.com/paper/2609.10883/) (2026)
- [SIRF: A Spec-Internalized Risk Foundation Model for Industrial Content Risk Control](https://mlsanity.com/paper/2609.11752/) (2026)
- [Shallow Beliefs: Synthetic document finetuning does not inoculate against emergent misalignment from reward hacking](https://mlsanity.com/paper/2609.14998/) (2026)
- [Constitutional adapters: Inference-time interventions for misalignment and misuse](https://mlsanity.com/paper/2609.36657/) (2026)
- [Character Training for Risk-Averse Agents](https://mlsanity.com/paper/2609.38093/) (2026)
- [Pre-training interventions, ex post facto: Grafting model beliefs across checkpoints](https://mlsanity.com/paper/2610.00767/) (2026)
- [SIGMA: Self-Improving Alignment Generalization from a Model Spec](https://mlsanity.com/paper/2610.07935/) (2026)
- [Predicting Alignment Generalization with Value Representations](https://mlsanity.com/paper/2610.12410/) (2026)

## Abstract (arXiv)

Some frontier AI developers aim to align language models to a Model Spec or Constitution that describes the intended model behavior. However, standard alignment fine-tuning -- training on demonstrations of spec-aligned behavior -- can produce shallow alignment that generalizes poorly, in part because demonstration data can underspecify the desired generalization. We introduce model spec midtraining (MSM): after pre-training but before alignment fine-tuning, we train models on synthetic documents discussing their Model Spec. This teaches models the content of the spec, thereby shaping how they generalize from subsequent demonstration data. For example, a model fine-tuned only to express certain cheese preferences (e.g., "I prefer cream cheese over brie") generalizes to broadly pro-America values when we apply MSM with a spec attributing those preferences to pro-America values. Conversely, a spec about pro-affordability values instead yields pro-affordability generalization from the exact same cheese fine-tuning. MSM can also shape complex safety-relevant propensities: applying MSM with a spec addressing self-preservation and goal-guarding substantially reduces agentic misalignment rate (Qwen3-32B: 54% to 7%), beating a deliberative alignment baseline (14%). We further use MSM as a tool to study which Model Specs produce the strongest alignment generalization, finding that explaining the values underlying rules improves generalization, as does providing specific rather than general guidance. Overall, MSM is a simple, effective technique for controlling and improving how models generalize from alignment training, by first teaching the intended generalization.