Beyond the Hard Budget: Sparsity Regularizers for More Interpretable Top-k Sparse Autoencoders

Sparse autoencoders (SAEs) have become a leading tool for interpreting the representations of vision foundation models, decomposing their polysemantic activations into a larger set of sparse, more monosemantic features. The Top-k SAE, a now-standard variant, enforces sparsity architecturally through its activation function, retaining only the k most active latents per input. Because it was designed precisely to avoid the ℓ1 penalty used by earlier SAEs and its known drawbacks, it has not been combined with an explicit sparsity regularizer. Yet the Top-k SAE retains limitations of its own, and we hypothesize that a sparsity penalty acting before the selection could sharpen each latent's selectivity and make the code more interpretable, without reintroducing the drawbacks of the ℓ1 penalty. We introduce two sparsity regularizers compatible with the Top-k architecture, both acting on the activations before the Top-k selection: an ℓ1 penalty on the unselected (off-support) units, and a scale-invariant ℓ1/ℓ2-ratio penalty that concentrates the code onto fewer effective units. Both penalties are applied only to the batch-active units, those selected by the Top-k operator at least once within the batch. Across two datasets, three vision foundation models, and a range of k, both regularizers consistently improve monosemanticity at no cost to reconstruction quality. The ℓ1/ℓ2 penalty further concentrates information into fewer latents, making reconstruction more robust to the inference-time choice of k and improving small-budget linear probing. Our central finding is that hard architectural sparsity and soft sparsity regularization are complementary rather than mutually exclusive.