# Sparse Coding with Multi-Layer Decoders using Variance Regularization

arXiv: https://arxiv.org/abs/2112.09214 · HTML: https://mlsanity.com/paper/2112.09214/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 13 of its 59 references that are in the map.
- First posted on arXiv 16 Dec 2021.

## Builds on (most cited first, 13 of 13)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Momentum Contrast for Unsupervised Visual Representation Learning](https://mlsanity.com/paper/1911.05722/) (2019)
- [Bootstrap your own latent: A new approach to self-supervised Learning](https://mlsanity.com/paper/2006.07733/) (2020)
- [Barlow Twins: Self-Supervised Learning via Redundancy Reduction](https://mlsanity.com/paper/2103.03230/) (2021)
- [VICReg: Variance-Invariance-Covariance Regularization for Self-Supervised Learning](https://mlsanity.com/paper/2105.04906/) (2021)
- [k-Sparse Autoencoders](https://mlsanity.com/paper/1312.5663/) (2013)
- [Difference Target Propagation](https://mlsanity.com/paper/1412.7525/) (2014)
- [A survey of sparse representation: algorithms and applications](https://mlsanity.com/paper/1602.07017/) (2016)
- [Sparse Modeling for Image and Vision Processing](https://mlsanity.com/paper/1411.3230/) (2014)
- [The Sparse Manifold Transform](https://mlsanity.com/paper/1806.08887/) (2018)
- [Deep Dictionary Learning: A PARametric NETwork Approach](https://mlsanity.com/paper/1803.04022/) (2018)
- [When Dictionary Learning Meets Deep Learning: Deep Dictionary Learning and Coding Network for Image Recognition with Limited Data](https://mlsanity.com/paper/2005.10940/) (2020)
- [Unsupervised Feature Learning by Deep Sparse Coding](https://mlsanity.com/paper/1312.5783/) (2013)

## Built on by (most cited first, 3 of 3)

- [LD-GAN: Low-Dimensional Generative Adversarial Network for Spectral Image Generation with Variance Regularization](https://mlsanity.com/paper/2305.00132/) (2023)
- [Learning Sparse Latent Representations for Generator Model](https://mlsanity.com/paper/2209.09949/) (2022)
- [Multi-Dictionary Learning for Low Rank Sparse Coding](https://mlsanity.com/paper/2509.10033/) (2025)

## Abstract (arXiv)

Sparse representations of images are useful in many computer vision applications. Sparse coding with an l1 penalty and a learned linear dictionary requires regularization of the dictionary to prevent a collapse in the l1 norms of the codes. Typically, this regularization entails bounding the Euclidean norms of the dictionary's elements. In this work, we propose a novel sparse coding protocol which prevents a collapse in the codes without the need to regularize the decoder. Our method regularizes the codes directly so that each latent code component has variance greater than a fixed threshold over a set of sparse representations for a given set of inputs. Furthermore, we explore ways to effectively train sparse coding systems with multi-layer decoders since they can model more complex relationships than linear dictionaries. In our experiments with MNIST and natural image patches, we show that decoders learned with our approach have interpretable features both in the linear and multi-layer case. Moreover, we show that sparse autoencoders with multi-layer decoders trained using our variance regularization method produce higher quality reconstructions with sparser representations when compared to autoencoders with linear dictionaries. Additionally, sparse representations obtained with our variance regularization approach are useful in the downstream tasks of denoising and classification in the low-data regime.