# Pivotal Auto-Encoder via Self-Normalizing ReLU

arXiv: https://arxiv.org/abs/2406.16052 · HTML: https://mlsanity.com/paper/2406.16052/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 19 of its 48 references that are in the map.
- First posted on arXiv 23 Jun 2024.

## Builds on (most cited first, 19 of 19)

- [Beyond a Gaussian Denoiser: Residual Learning of Deep CNN for Image Denoising](https://mlsanity.com/paper/1608.03981/) (2016)
- [FFDNet: Toward a Fast and Flexible Solution for CNN based Image Denoising](https://mlsanity.com/paper/1710.04026/) (2017)
- [Algorithm Unrolling: Interpretable, Efficient Deep Learning for Signal and Image Processing](https://mlsanity.com/paper/1912.10557/) (2019)
- [Deep Unfolding Network for Image Super-Resolution](https://mlsanity.com/paper/2003.10428/) (2020)
- [Robust and interpretable blind image denoising via bias-free convolutional neural networks](https://mlsanity.com/paper/1906.05478/) (2019)
- [Convolutional Neural Networks Analyzed via Convolutional Sparse Coding](https://mlsanity.com/paper/1607.08194/) (2016)
- [On Multi-Layer Basis Pursuit, Efficient Algorithms and Convolutional Neural Networks](https://mlsanity.com/paper/1806.00701/) (2018)
- [Rethinking the CSC Model for Natural Images](https://mlsanity.com/paper/1909.05742/) (2019)
- [SGD-Net: Efficient Model-Based Deep Learning with Theoretical Guarantees](https://mlsanity.com/paper/2101.09379/) (2021)
- [Training deep learning based image denoisers from undersampled measurements without ground truth and without image prior](https://mlsanity.com/paper/1806.00961/) (2018)
- [Multi-Layer Convolutional Sparse Modeling: Pursuit and Dictionary Learning](https://mlsanity.com/paper/1708.08705/) (2017)
- [Boosting of Image Denoising Algorithms](https://mlsanity.com/paper/1502.06220/) (2015)
- [VIDOSAT: High-dimensional Sparsifying Transform Learning for Online Video Denoising](https://mlsanity.com/paper/1710.00947/) (2017)
- [One Size Fits All: Can We Train One Denoiser for All Noise Levels?](https://mlsanity.com/paper/2005.09627/) (2020)
- [Multi-Layer Sparse Coding: The Holistic Way](https://mlsanity.com/paper/1804.09788/) (2018)
- [Adversarial Noise Attacks of Deep Learning Architectures -- Stability Analysis via Sparse Modeled Signals](https://mlsanity.com/paper/1805.11596/) (2018)
- [FRIST - Flipping and Rotation Invariant Sparsifying Transform Learning and Applications](https://mlsanity.com/paper/1511.06359/) (2015)
- [Support recovery and sup-norm convergence rates for sparse pivotal estimation](https://mlsanity.com/paper/2001.05401/) (2020)
- [Noise-Level Estimation from Single Color Image Using Correlations Between Textures in RGB Channels](https://mlsanity.com/paper/1904.02566/) (2019)

## Built on by (most cited first, 1 of 1)

- [Don't Fear Peculiar Activation Functions: EUAF and Beyond](https://mlsanity.com/paper/2407.09580/) (2024)

## Abstract (arXiv)

Sparse auto-encoders are useful for extracting low-dimensional representations from high-dimensional data. However, their performance degrades sharply when the input noise at test time differs from the noise employed during training. This limitation hinders the applicability of auto-encoders in real-world scenarios where the level of noise in the input is unpredictable. In this paper, we formalize single hidden layer sparse auto-encoders as a transform learning problem. Leveraging the transform modeling interpretation, we propose an optimization problem that leads to a predictive model invariant to the noise level at test time. In other words, the same pre-trained model is able to generalize to different noise levels. The proposed optimization algorithm, derived from the square root lasso, is translated into a new, computationally efficient auto-encoding architecture. After proving that our new method is invariant to the noise level, we evaluate our approach by training networks using the proposed architecture for denoising tasks. Our experimental results demonstrate that the trained models yield a significant improvement in stability against varying types of noise compared to commonly used architectures.