# Saturating Auto-Encoders

arXiv: https://arxiv.org/abs/1301.3577 · HTML: https://mlsanity.com/paper/1301.3577/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 16 Jan 2013.

## Built on by (most cited first, 3 of 3)

- [Piecewise Linear Units Improve Deep Neural Networks](https://mlsanity.com/paper/2108.00700/) (2021)
- [Enhancing Classifier Conservativeness and Robustness by Polynomiality](https://mlsanity.com/paper/2203.12693/) (2022)
- [Dynamic-Deep: Tune ECG Task Performance and Optimize Compression in IoT Architectures](https://mlsanity.com/paper/2106.00606/) (2021)

## Abstract (arXiv)

We introduce a simple new regularizer for auto-encoders whose hidden-unit activation functions contain at least one zero-gradient (saturated) region. This regularizer explicitly encourages activations in the saturated region(s) of the corresponding activation function. We call these Saturating Auto-Encoders (SATAE). We show that the saturation regularizer explicitly limits the SATAE's ability to reconstruct inputs which are not near the data manifold. Furthermore, we show that a wide variety of features can be learned when different activation functions are used. Finally, connections are established with the Contractive and Sparse Auto-Encoders.