# Automatically Finding Rule-Based Neurons in OthelloGPT

arXiv: https://arxiv.org/abs/2511.00059 · HTML: https://mlsanity.com/paper/2511.00059/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 5 of its 9 references that are in the map.
- First posted on arXiv 28 Oct 2025.

## Builds on (most cited first, 5 of 5)

- [Toy Models of Superposition](https://mlsanity.com/paper/2209.10652/) (2022)
- [Emergent World Representations: Exploring a Sequence Model Trained on a Synthetic Task](https://mlsanity.com/paper/2210.13382/) (2022)
- [Emergent Linear Representations in World Models of Self-Supervised Sequence Models](https://mlsanity.com/paper/2309.00941/) (2023)
- [Measuring Progress in Dictionary Learning for Language Model Interpretability with Board Game Models](https://mlsanity.com/paper/2408.00113/) (2024)
- [Linear Latent World Models in Simple Transformers: A Case Study on Othello-GPT](https://mlsanity.com/paper/2310.07582/) (2023)

## Abstract (arXiv)

OthelloGPT, a transformer trained to predict valid moves in Othello, provides an ideal testbed for interpretability research. The model is complex enough to exhibit rich computational patterns, yet grounded in rule-based game logic that enables meaningful reverse-engineering. We present an automated approach based on decision trees to identify and interpret MLP neurons that encode rule-based game logic. Our method trains regression decision trees to map board states to neuron activations, then extracts decision paths where neurons are highly active to convert them into human-readable logical forms. These descriptions reveal highly interpretable patterns; for instance, neurons that specifically detect when diagonal moves become legal. Our findings suggest that roughly half of the neurons in layer 5 can be accurately described by compact, rule-based decision trees (R2 > 0.7 for 913 of 2,048 neurons), while the remainder likely participate in more distributed or non-rule-based computations. We verify the causal relevance of patterns identified by our decision trees through targeted interventions. For a specific square, for specific game patterns, we ablate neurons corresponding to those patterns and find an approximately 5-10 fold stronger degradation in the model's ability to predict legal moves along those patterns compared to control patterns. To facilitate future work, we provide a Python tool that maps rule-based game behaviors to their implementing neurons, serving as a resource for researchers to test whether their interpretability methods recover meaningful computational structures.