# Mimetic Initialization of MLPs

arXiv: https://arxiv.org/abs/2602.07156 · HTML: https://mlsanity.com/paper/2602.07156/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 15 of its 16 references that are in the map.
- First posted on arXiv 6 Feb 2026.

## Builds on (most cited first, 15 of 15)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift](https://mlsanity.com/paper/1502.03167/) (2015)
- [Layer Normalization](https://mlsanity.com/paper/1607.06450/) (2016)
- [Training data-efficient image transformers & distillation through attention](https://mlsanity.com/paper/2012.12877/) (2020)
- [A ConvNet for the 2020s](https://mlsanity.com/paper/2201.03545/) (2022)
- [Big Transfer (BiT): General Visual Representation Learning](https://mlsanity.com/paper/1912.11370/) (2019)
- [Patches Are All You Need?](https://mlsanity.com/paper/2201.09792/) (2022)
- [Unveiling Transformers with LEGO: a synthetic reasoning task](https://mlsanity.com/paper/2206.04301/) (2022)
- [Mimetic Initialization of Self-Attention Layers](https://mlsanity.com/paper/2305.09828/) (2023)
- [Get rich quick: exact solutions reveal how unbalanced initializations promote rapid feature learning](https://mlsanity.com/paper/2406.06158/) (2024)
- [On the Surprising Effectiveness of Attention Transfer for Vision Transformers](https://mlsanity.com/paper/2411.09702/) (2024)
- [Understanding the Covariance Structure of Convolutional Filters](https://mlsanity.com/paper/2210.03651/) (2022)
- [Mimetic Initialization Helps State Space Models Learn to Recall](https://mlsanity.com/paper/2410.11135/) (2024)
- [Convolutional Initialization for Data-Efficient Vision Transformers](https://mlsanity.com/paper/2401.12511/) (2024)

## Built on by (most cited first, 2 of 2)

- [Attention Transfer Is Not Universally Effective for Vision Transformers](https://mlsanity.com/paper/2605.07191/) (2026)
- [Copy the Same, Distill the Difference: Initializing Linear Vision Transformers](https://mlsanity.com/paper/2609.35745/) (2026)

## Abstract (arXiv)

Mimetic initialization uses pretrained models as case studies of good initialization, using observations of structures in trained weights to inspire new, simple initialization techniques. So far, it has been applied only to spatial mixing layers, such convolutional, self-attention, and state space layers. In this work, we present the first attempt to apply the method to channel mixing layers, namely multilayer perceptrons (MLPs). Our extremely simple technique for MLPs -- to give the first layer a nonzero mean -- speeds up training on small-scale vision tasks like CIFAR-10 and ImageNet-1k. Though its effect is much smaller than spatial mixing initializations, it can be used in conjunction with them for an additional positive effect.