# Simple LLM Baselines are Competitive for Model Diffing

arXiv: https://arxiv.org/abs/2602.10371 · HTML: https://mlsanity.com/paper/2602.10371/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Placed in SAE features and their applications, one of 13 directions in Mechanistic interpretability.
- Builds on 11 of its 23 references that are in the map.
- First posted on arXiv 10 Feb 2026.

## Placed in

- Mechanistic interpretability / [SAE features and their applications](https://mlsanity.com/mechinterp/sae-features-and-their-applications/)

## Builds on (most cited first, 11 of 11)

- [Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena](https://mlsanity.com/paper/2306.05685/) (2023)
- [Gemini 2.5: Pushing the Frontier with Advanced Reasoning, Multimodality, Long Context, and Next Generation Agentic Capabilities](https://mlsanity.com/paper/2507.06261/) (2025)
- [UMAP: Uniform Manifold Approximation and Projection for Dimension Reduction](https://mlsanity.com/paper/1802.03426/) (2018)
- [Sparse Autoencoders Find Highly Interpretable Features in Language Models](https://mlsanity.com/paper/2309.08600/) (2023)
- [WildChat: 1M ChatGPT Interaction Logs in the Wild](https://mlsanity.com/paper/2405.01470/) (2024)
- [Sleeper Agents: Training Deceptive LLMs that Persist Through Safety Training](https://mlsanity.com/paper/2401.05566/) (2024)
- [Model Organisms for Emergent Misalignment](https://mlsanity.com/paper/2506.11613/) (2025)
- [Llama-Embed-Nemotron-8B: A Universal Text Embedding Model for Multilingual and Cross-Lingual Tasks](https://mlsanity.com/paper/2511.07025/) (2025)
- [Eliciting Secret Knowledge from Language Models](https://mlsanity.com/paper/2510.01070/) (2025)
- [VibeCheck: Discover and Quantify Qualitative Differences in Large Language Models](https://mlsanity.com/paper/2410.12851/) (2024)
- [Interpretable Embeddings with Sparse Autoencoders: A Data Analysis Toolkit](https://mlsanity.com/paper/2512.10092/) (2025)

## Built on by (most cited first, 3 of 3)

- [How Transparent is DiffusionGemma?](https://mlsanity.com/paper/2606.20560/) (2026)
- [Most Current Model Organisms Are Leaky: Perplexity Differencing Often Reveals Finetuning Objectives](https://mlsanity.com/paper/2605.00994/) (2026)
- [Size Doesn't Matter: Cosine-Scored Sparse Autoencoders](https://mlsanity.com/paper/2606.15054/) (2026)

## Abstract (arXiv)

Standard LLM evaluations only test capabilities or dispositions that evaluators designed them for, missing unexpected differences such as behavioral shifts between model revisions or emergent misaligned tendencies. Model diffing addresses this limitation by automatically surfacing systematic behavioral differences. Recent approaches include LLM-based methods that generate natural language descriptions and sparse autoencoder (SAE)-based methods that identify interpretable features. However, no systematic comparison of these approaches exists nor are there established evaluation criteria. We address this gap by proposing evaluation metrics for key desiderata (generalization, interestingness, and abstraction level) and use these to compare existing methods. Our results show that an improved LLM-based baseline performs comparably to the SAE-based method while typically surfacing more abstract behavioral differences.