# SOSAE: Self-Organizing Sparse AutoEncoder

arXiv: https://arxiv.org/abs/2507.04644 · HTML: https://mlsanity.com/paper/2507.04644/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Placed in Sparse autoencoder training and evaluation, one of 13 directions in Mechanistic interpretability.
- Builds on 7 of its 34 references that are in the map.
- First posted on arXiv 7 Jul 2025.

## Placed in

- Mechanistic interpretability / [Sparse autoencoder training and evaluation](https://mlsanity.com/mechinterp/sparse-autoencoder-training-and-evaluation/)

## Builds on (most cited first, 7 of 7)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [GraphVAE: Towards Generation of Small Graphs Using Variational Autoencoders](https://mlsanity.com/paper/1802.03480/) (2018)
- [k-Sparse Autoencoders](https://mlsanity.com/paper/1312.5663/) (2013)
- [Variational Autoencoder for Deep Learning of Images, Labels and Captions](https://mlsanity.com/paper/1609.08976/) (2016)
- [Generalized Denoising Auto-Encoders as Generative Models](https://mlsanity.com/paper/1305.6663/) (2013)
- [An Additive Autoencoder for Dimension Estimation](https://mlsanity.com/paper/2210.06773/) (2022)
- [Learning Sparse Filters in Deep Convolutional Neural Networks with a l1/l2 Pseudo-Norm](https://mlsanity.com/paper/2007.10022/) (2020)

## Abstract (arXiv)

The process of tuning the size of the hidden layers for autoencoders has the benefit of providing optimally compressed representations for the input data. However, such hyper-parameter tuning process would take a lot of computation and time effort with grid search as the default option. In this paper, we introduce the Self-Organization Regularization for Autoencoders that dynamically adapts the dimensionality of the feature space to the optimal size. Inspired by physics concepts, Self-Organizing Sparse AutoEncoder (SOSAE) induces sparsity in feature space in a structured way that permits the truncation of the non-active part of the feature vector without any loss of information. This is done by penalizing the autoencoder based on the magnitude and the positional index of the feature vector dimensions, which during training constricts the feature space in both terms. Extensive experiments on various datasets show that our SOSAE can tune the feature space dimensionality up to 130 times lesser Floating-point Operations (FLOPs) than other baselines while maintaining the same quality of tuning and performance.