# Generalizing MLPs With Dropouts, Batch Normalization, and Skip Connections

arXiv: https://arxiv.org/abs/2108.08186 · HTML: https://mlsanity.com/paper/2108.08186/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 11 of its 24 references that are in the map.
- First posted on arXiv 18 Aug 2021.

## Builds on (most cited first, 11 of 11)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift](https://mlsanity.com/paper/1502.03167/) (2015)
- [Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learning](https://mlsanity.com/paper/1506.02142/) (2015)
- [Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classification](https://mlsanity.com/paper/1502.01852/) (2015)
- [A simple yet effective baseline for 3d human pose estimation](https://mlsanity.com/paper/1705.03098/) (2017)
- [Tune: A Research Platform for Distributed Model Selection and Training](https://mlsanity.com/paper/1807.05118/) (2018)
- [Compacting, Picking and Growing for Unforgetting Continual Learning](https://mlsanity.com/paper/1910.06562/) (2019)
- [Fine-Grained Age Estimation in the wild with Attention LSTM Networks](https://mlsanity.com/paper/1805.10445/) (2018)
- [Rethinking the Usage of Batch Normalization and Dropout in the Training of Deep Neural Networks](https://mlsanity.com/paper/1905.05928/) (2019)

## Built on by (most cited first, 3 of 3)

- [MiVOLO: Multi-input Transformer for Age and Gender Estimation](https://mlsanity.com/paper/2307.04616/) (2023)
- [P-Age: Pexels Dataset for Robust Spatio-Temporal Apparent Age Classification](https://mlsanity.com/paper/2311.02432/) (2023)
- [A Hybrid Transformer-Sequencer approach for Age and Gender classification from in-wild facial images](https://mlsanity.com/paper/2403.12483/) (2024)

## Abstract (arXiv)

A multilayer perceptron (MLP) is typically made of multiple fully connected layers with nonlinear activation functions. There have been several approaches to make them better (e.g. faster convergence, better convergence limit, etc.). But the researches lack structured ways to test them. We test different MLP architectures by carrying out the experiments on the age and gender datasets. We empirically show that by whitening inputs before every linear layer and adding skip connections, our proposed MLP architecture can result in better performance. Since the whitening process includes dropouts, it can also be used to approximate Bayesian inference. We have open sourced our code, and released models and docker images at https://github.com/tae898/age-gender/