# From Dataset Spectral Geometry to Network Weights: A Geometry-Aware Initialization for Sigmoidal MLPs in Image Classification

arXiv: https://arxiv.org/abs/2606.28444 · HTML: https://mlsanity.com/paper/2606.28444/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 6 of its 20 references that are in the map.
- First posted on arXiv 26 Jun 2026.

## Builds on (most cited first, 6 of 6)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithms](https://mlsanity.com/paper/1708.07747/) (2017)
- [Tropical Geometry of Deep Neural Networks](https://mlsanity.com/paper/1805.07091/) (2018)
- [From Universal Approximation Theorem to Tropical Geometry of Multi-Layer Perceptrons](https://mlsanity.com/paper/2510.15012/) (2025)

## Abstract (arXiv)

Classical universal approximation theorems (UAT) establish the expressive power of sigmoidal multilayer perceptrons, but they do not specify how the weights should be initialized. We study a supervised, data-dependent, geometry-aware initialization for one-hidden-layer sigmoidal MLPs that compiles labeled class geometry into network weights. The construction starts from the idea that sigmoid units can act as smooth half-space gates. For each class, we center the training samples at their mean, apply SVD to estimate principal directions and spectral scales, select retained directions by an energy threshold, and represent each retained direction by a pair of sigmoid slab gates. These class-specific gates are then concatenated into a shared hidden layer initialized directly from the training set. We also formulate a SVD-Mahalanobis subspace classifier as a non-neural geometric reference, which tests whether the estimated spectral class geometry is already discriminative before being embedded into the MLP. Experiments on MNIST, Fashion-MNIST, and CIFAR-10 show that the proposed initializer produces a substantially more informative zero-epoch state than a matched task-agnostic Xavier reference, while full training reaches comparable final accuracy. Frozen-hidden experiments and neutral-head ablations further show that the class-wise SVD gates remain useful fixed features, even after the initially aligned output layer is replaced by a Xavier-initialized one.