# The Model Organism Lottery: Model Organism Interpretability Strongly Depends on Training Methodology

arXiv: https://arxiv.org/abs/2607.01033 · HTML: https://mlsanity.com/paper/2607.01033/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- Placed in Knowledge and representations inside LLMs, one of 13 directions in Mechanistic interpretability.
- Builds on 13 of its 23 references that are in the map.
- First posted on arXiv 1 Jul 2026.

## Placed in

- Mechanistic interpretability / [Knowledge and representations inside LLMs](https://mlsanity.com/mechinterp/knowledge-and-representations-inside-llms/)

## Builds on (most cited first, 13 of 13)

- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [Direct Preference Optimization: Your Language Model is Secretly a Reward Model](https://mlsanity.com/paper/2305.18290/) (2023)
- [Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback](https://mlsanity.com/paper/2204.05862/) (2022)
- [Sparse Autoencoders Find Highly Interpretable Features in Language Models](https://mlsanity.com/paper/2309.08600/) (2023)
- [WizardLM: Empowering large pre-trained language models to follow complex instructions](https://mlsanity.com/paper/2304.12244/) (2023)
- [Steering Language Models With Activation Engineering](https://mlsanity.com/paper/2308.10248/) (2023)
- [Sleeper Agents: Training Deceptive LLMs that Persist Through Safety Training](https://mlsanity.com/paper/2401.05566/) (2024)
- [AI Sandbagging: Language Models can Strategically Underperform on Evaluations](https://mlsanity.com/paper/2406.07358/) (2024)
- [Auditing language models for hidden objectives](https://mlsanity.com/paper/2503.10965/) (2025)
- [HelpSteer3-Preference: Open Human-Annotated Preference Data across Diverse Tasks and Languages](https://mlsanity.com/paper/2505.11475/) (2025)
- [Activation Oracles: Training and Evaluating LLMs as General-Purpose Activation Explainers](https://mlsanity.com/paper/2512.15674/) (2025)
- [Narrow Finetuning Leaves Clearly Readable Traces in Activation Differences](https://mlsanity.com/paper/2510.13900/) (2025)
- [Towards eliciting latent knowledge from LLMs with mechanistic interpretability](https://mlsanity.com/paper/2505.14352/) (2025)

## Built on by (most cited first, 6 of 6)

- [Hidden APIs in Language Models: Discovering Reusable Causal Interfaces from Forked Futures](https://mlsanity.com/paper/2607.27617/) (2026)
- [Would this change your answer? Evaluating Explanations of LLM Behavior In The Wild with Counterfactual Experiments](https://mlsanity.com/paper/2608.16747/) (2026)
- [Activation Flow: Manufacturing Activations for Steering](https://mlsanity.com/paper/2609.32530/) (2026)
- [Pre-training interventions, ex post facto: Grafting model beliefs across checkpoints](https://mlsanity.com/paper/2610.00767/) (2026)
- [How to train your model organism](https://mlsanity.com/paper/2610.10203/) (2026)
- [On the Monitorability of Natural Language Autoencoders: Identification vs Discrimination](https://mlsanity.com/paper/2610.18727/) (2026)

## Abstract (arXiv)

Model organisms (MOs) - language models trained to exhibit undesired or unnatural behaviours - are frequently used as testbeds for evaluating white-box interpretability techniques. Current MOs are typically constructed via post-hoc supervised fine-tuning (SFT) on behavioural transcripts or synthetic documents. Prior research has shown that interpretability methods can easily identify hidden behaviours in these MOs. However, recent work suggests that such post-hoc training methods may make interpretability unrealistically easy. We investigate this claim by constructing a suite of 54 verb|OLMo2-1B|- and verb|gemma-3-1b-it|-based MOs trained with seven different techniques, including standard post-hoc SFT, post-hoc DPO, and more realistic integration of MO data into the OLMo post-training DPO phase. We use these MO variants to benchmark activation oracles, activation steering, logit lens, and sparse autoencoders. Our findings show that (i) MO interpretability depends strongly on training objective, target behaviour, model architecture, and training data generation pipeline; (ii) substantial variance remains even after controlling for differences in the strength of target behaviour expression; and (iii) our more realistic textitintegrated training often yields less interpretable MOs than standard post-hoc methods. Our results cast substantial doubt on the validity of current MOs as interpretability proxies.