# Predicting how model propensities generalize out of distribution

arXiv: https://arxiv.org/abs/2610.14937 · HTML: https://mlsanity.com/paper/2610.14937/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 14 of its 20 references that are in the map.
- First posted on arXiv 10 Oct 2026.

## Builds on (most cited first, 14 of 14)

- [Shortcut Learning in Deep Neural Networks](https://mlsanity.com/paper/2004.07780/) (2020)
- [Fine-tuning Aligned Language Models Compromises Safety, Even When Users Do Not Intend To!](https://mlsanity.com/paper/2310.03693/) (2023)
- [The Reversal Curse: LLMs trained on "A is B" fail to learn "B is A"](https://mlsanity.com/paper/2309.12288/) (2023)
- [Emergent Misalignment: Narrow finetuning can produce broadly misaligned LLMs](https://mlsanity.com/paper/2502.17424/) (2025)
- [Subliminal Learning: Language models transmit behavioral traits via hidden signals in data](https://mlsanity.com/paper/2507.14805/) (2025)
- [Weird Generalization and Inductive Backdoors: New Ways to Corrupt LLMs](https://mlsanity.com/paper/2512.09742/) (2025)
- [Inoculation Prompting: Eliciting traits from LLMs during training can suppress them at test-time](https://mlsanity.com/paper/2510.04340/) (2025)
- [Inoculation Prompting: Instructing LLMs to misbehave at train-time improves test-time alignment](https://mlsanity.com/paper/2510.05024/) (2025)
- [Steering Out-of-Distribution Generalization with Concept Ablation Fine-Tuning](https://mlsanity.com/paper/2507.16795/) (2025)
- [Narrow Finetuning Leaves Clearly Readable Traces in Activation Differences](https://mlsanity.com/paper/2510.13900/) (2025)
- [Chunky Post-Training: Data Driven Failures of Generalization](https://mlsanity.com/paper/2602.05910/) (2026)
- [Emergently Misaligned Language Models Show Behavioral Self-Awareness That Shifts With Subsequent Realignment](https://mlsanity.com/paper/2602.14777/) (2026)
- [Negation Neglect: When models fail to learn negations in training](https://mlsanity.com/paper/2605.13829/) (2026)
- [The Consciousness Cluster: Emergent preferences of Models that Claim to be Conscious](https://mlsanity.com/paper/2604.13051/) (2026)

## Abstract (arXiv)

How do language models generalize from their training data? This is a central challenge for alignment: models reliably learn their training distribution given enough samples, but we want models that behave well when deployed in new contexts. We study out-of-distribution (OOD) generalization of propensities. Our central experiment uses a range of elicitation techniques -- system prompting, few-shot learning, SFT on reference answers, and GRPO -- to increase a target propensity in a model, then measures how it spills over onto others across the resulting cross-elicitation matrices. We find that spillover is highly correlated between models when the same elicitation technique is used, but our SFT and GRPO runs induce different spillover profiles. Elicitation of benign traits can sometimes generalize towards undesired traits, for example a model trained to claim that it is sentient often becomes more power-seeking. We further find that spillover partially resembles trait correlations found in humans and identify three clusters of traits that account for about half of the variance using exploratory factor analysis (EFA). Across many elicitation runs, we observe that models change in characteristic ways that don't depend on the targeted trait, a phenomenon we refer to as drift. Drift moves Qwen3.5-9B towards its base model. When investigating if LLMs are able to predict spillover, we find that Claude Opus 5.5 predicts spillover slightly better than the models themselves, but worse than data-driven approaches using alternative elicitation methods or models. Overall, no single dominant predictor emerges, and generalization remains hard to predict. Code and data are available at https://github.com/nielsrolf/propensity-spillover.