# Emergent Hierarchical Monosemantic Neurons from the Group-Contrastive Forward-Forward Algorithm

arXiv: https://arxiv.org/abs/2607.16295 · HTML: https://mlsanity.com/paper/2607.16295/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Placed in SAE features and their applications, one of 13 directions in Mechanistic interpretability.
- Builds on 27 of its 38 references that are in the map.
- First posted on arXiv 13 Jul 2026.

## Placed in

- Mechanistic interpretability / [SAE features and their applications](https://mlsanity.com/mechinterp/sae-features-and-their-applications/)

## Builds on (most cited first, 27 of 27)

- [Sparse Autoencoders Find Highly Interpretable Features in Language Models](https://mlsanity.com/paper/2309.08600/) (2023)
- [The Mythos of Model Interpretability](https://mlsanity.com/paper/1606.03490/) (2016)
- [The Linear Representation Hypothesis and the Geometry of Large Language Models](https://mlsanity.com/paper/2311.03658/) (2023)
- [Scaling and evaluating sparse autoencoders](https://mlsanity.com/paper/2406.04093/) (2024)
- [Mechanistic Interpretability for AI Safety -- A Review](https://mlsanity.com/paper/2404.14082/) (2024)
- [The Forward-Forward Algorithm: Some Preliminary Investigations](https://mlsanity.com/paper/2212.13345/) (2022)
- [Not All Language Model Features Are One-Dimensionally Linear](https://mlsanity.com/paper/2405.14860/) (2024)
- [Transcoders Find Interpretable LLM Feature Circuits](https://mlsanity.com/paper/2406.11944/) (2024)
- [Scaling Monosemanticity: Extracting Interpretable Features from Claude 3 Sonnet](https://mlsanity.com/paper/2605.29358/) (2026)
- [The Geometry of Categorical and Hierarchical Concepts in Large Language Models](https://mlsanity.com/paper/2406.01506/) (2024)
- [Learning Multi-Level Features with Matryoshka Sparse Autoencoders](https://mlsanity.com/paper/2503.17547/) (2025)
- [Neural population geometry: An approach for understanding biological and artificial neural networks](https://mlsanity.com/paper/2104.07059/) (2021)
- [SymBa: Symmetric Backpropagation-Free Contrastive Learning with Forward-Forward Algorithm for Optimizing Convergence](https://mlsanity.com/paper/2303.08418/) (2023)
- [Transcoders Beat Sparse Autoencoders for Interpretability](https://mlsanity.com/paper/2501.18823/) (2025)
- [Layer Collaboration in the Forward-Forward Algorithm](https://mlsanity.com/paper/2305.12393/) (2023)
- [The Cascaded Forward Algorithm for Neural Network Training](https://mlsanity.com/paper/2303.09728/) (2023)
- [Self-Contrastive Forward-Forward Algorithm](https://mlsanity.com/paper/2409.11593/) (2024)
- [Emergent representations in networks trained with the Forward-Forward algorithm](https://mlsanity.com/paper/2305.18353/) (2023)
- [Rep2Text: Decoding Full Text from a Single LLM Token Representation](https://mlsanity.com/paper/2511.06571/) (2025)
- [Human-like Content Analysis for Generative AI with Language-Grounded Sparse Encoders](https://mlsanity.com/paper/2508.18236/) (2025)
- [BrainExplore: Large-Scale Discovery of Interpretable Visual Representations in the Human Brain](https://mlsanity.com/paper/2512.08560/) (2025)
- [The Integrated Forward-Forward Algorithm: Integrating Forward-Forward and Shallow Backpropagation With Local Losses](https://mlsanity.com/paper/2305.12960/) (2023)
- [How does My Model Fail? Automatic Identification and Interpretation of Physical Plausibility Failure Modes with Matryoshka Transcoders](https://mlsanity.com/paper/2511.10094/) (2025)
- [A Unified Theory of Sparse Dictionary Learning in Mechanistic Interpretability: Piecewise Biconvexity and Spurious Minima](https://mlsanity.com/paper/2512.05534/) (2025)
- [Bridging Mechanistic Interpretability and Prompt Engineering with Gradient Ascent for Interpretable Persona Control](https://mlsanity.com/paper/2601.02896/) (2026)
- [CXR-LanIC: Language-Grounded Interpretable Classifier for Chest X-Ray Diagnosis](https://mlsanity.com/paper/2510.21464/) (2025)
- [Toward Identifiable Sparse Autoencoders](https://mlsanity.com/paper/2605.31245/) (2026)

## Abstract (arXiv)

Mechanistic interpretability has made significant strides in understanding neural network representations, with sparse dictionary learning (SDL) methods, most prominently sparse autoencoders, as a central paradigm. However, recent work has reported several limitations of this paradigm: SDL objectives are non-identifiable; SDL methods rely heavily on the Linear Representation Hypothesis; and a growing body of evidence points to concepts that are encoded non-linearly and are therefore not expressible as any single direction. We hypothesise that a different route to monosemanticity is available. Biological visual systems exhibit highly selective neurons organised into hierarchies of increasing abstraction, and this organisation emerges from local, layer-wise learning rules rather than from a global error signal; we therefore ask whether a biologically plausible learning algorithm will likewise yield monosemantic neurons. To test this, we propose Group-Contrastive Forward-Forward (GCFF), a forward-forward training algorithm that combines class-specific routing with within-class contrastive objectives, reaching monosemanticity through architectural constraints rather than sparsity. Because GCFF attaches multiple non-linear layers to the representation under study, its neurons can therefore capture the non-linear concepts. On CLIP representations, a single trained GCFF module recovers monosemantic neurons whose abstraction increases progressively with depth, reaching environmental properties that hold independently of an image's foreground, without any sparsity constraint or supervision of abstraction level. We further demonstrate that GCFF can train networks from scratch, achieving state-of-the-art performance among forward-forward algorithms on various image classification benchmarks.