# Hedonic Neurons: A Mechanistic Mapping of Latent Coalitions in Transformer MLPs

arXiv: https://arxiv.org/abs/2509.23684 · HTML: https://mlsanity.com/paper/2509.23684/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 21 of its 39 references that are in the map.
- First posted on arXiv 28 Sep 2025.

## Builds on (most cited first, 21 of 21)

- [The Llama 3 Herd of Models](https://mlsanity.com/paper/2407.21783/) (2024)
- [Pythia: A Suite for Analyzing Large Language Models Across Training and Scaling](https://mlsanity.com/paper/2304.01373/) (2023)
- [MS MARCO: A Human Generated MAchine Reading COmprehension Dataset](https://mlsanity.com/paper/1611.09268/) (2016)
- [Sparse Autoencoders Find Highly Interpretable Features in Language Models](https://mlsanity.com/paper/2309.08600/) (2023)
- [Transformer Feed-Forward Layers Are Key-Value Memories](https://mlsanity.com/paper/2012.14913/) (2020)
- [Toy Models of Superposition](https://mlsanity.com/paper/2209.10652/) (2022)
- [Knowledge Neurons in Pretrained Transformers](https://mlsanity.com/paper/2104.08696/) (2021)
- [Finding Neurons in a Haystack: Case Studies with Sparse Probing](https://mlsanity.com/paper/2305.01610/) (2023)
- [Overview of the TREC 2019 deep learning track](https://mlsanity.com/paper/2003.07820/) (2020)
- [Fine-Tuning LLaMA for Multi-Stage Text Retrieval](https://mlsanity.com/paper/2310.08319/) (2023)
- [Scaling Deep Contrastive Learning Batch Size under Memory Limited Setup](https://mlsanity.com/paper/2101.06983/) (2021)
- [Introducing LETOR 4.0 Datasets](https://mlsanity.com/paper/1306.2597/) (2013)
- [Scan and Snap: Understanding Training Dynamics and Token Composition in 1-layer Transformer](https://mlsanity.com/paper/2305.16380/) (2023)
- [Tevatron 2.0: Unified Document Retrieval Toolkit across Scale, Language, and Modality](https://mlsanity.com/paper/2505.02466/) (2025)
- [Bilinear MLPs enable weight-based mechanistic interpretability](https://mlsanity.com/paper/2410.08417/) (2024)
- [An Empirical Study on Parameter-Efficient Fine-Tuning for MultiModal Large Language Models](https://mlsanity.com/paper/2406.05130/) (2024)
- [Mechanistic Permutability: Match Features Across Layers](https://mlsanity.com/paper/2410.07656/) (2024)
- [Stochastic Parameter Decomposition](https://mlsanity.com/paper/2506.20790/) (2025)
- [NeurFlow: Interpreting Neural Networks through Neuron Groups and Functional Interactions](https://mlsanity.com/paper/2502.16105/) (2025)
- [How Relevance Emerges: Interpreting LoRA Fine-Tuning in Reranking LLMs](https://mlsanity.com/paper/2504.08780/) (2025)
- [Probing Ranking LLMs: A Mechanistic Analysis for Information Retrieval](https://mlsanity.com/paper/2410.18527/) (2024)

## Built on by (most cited first, 1 of 1)

- [The Commit-Abstain Circuit: Why Language Models Hallucinate Instead of Abstaining](https://mlsanity.com/paper/2609.32964/) (2026)

## Abstract (arXiv)

Fine-tuned Large Language Models (LLMs) encode rich task-specific features, but the form of these representations, especially within MLP layers, remains unclear. Empirical inspection of LoRA updates shows that new features concentrate in mid-layer MLPs, yet the scale of these layers obscures meaningful structure. Prior probing suggests that statistical priors may strengthen, split, or vanish across depth, motivating the need to study how neurons work together rather than in isolation. We introduce a mechanistic interpretability framework based on coalitional game theory, where neurons mimic agents in a hedonic game whose preferences capture their synergistic contributions to layer-local computations. Using top-responsive utilities and the PAC-Top-Cover algorithm, we extract stable coalitions of neurons: groups whose joint ablation has non-additive effects. We then track their transitions across layers as persistence, splitting, merging, or disappearance. Applied to LLaMA, Mistral, and Pythia rerankers fine-tuned on scalar IR tasks, our method finds coalitions with consistently higher synergy than clustering baselines. By revealing how neurons cooperate to encode features, hedonic coalitions uncover higher-order structure beyond disentanglement and yield computational units that are functionally important, interpretable, and predictive across domains.