# Variational Sparse Coding with Learned Thresholding

arXiv: https://arxiv.org/abs/2205.03665 · HTML: https://mlsanity.com/paper/2205.03665/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- Builds on 24 of its 61 references that are in the map.
- First posted on arXiv 7 May 2022.

## Builds on (most cited first, 24 of 24)

- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [Neural Discrete Representation Learning](https://mlsanity.com/paper/1711.00937/) (2017)
- [Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithms](https://mlsanity.com/paper/1708.07747/) (2017)
- [Deep Learning Face Attributes in the Wild](https://mlsanity.com/paper/1411.7766/) (2014)
- [Categorical Reparameterization with Gumbel-Softmax](https://mlsanity.com/paper/1611.01144/) (2016)
- [Estimating or Propagating Gradients Through Stochastic Neurons for Conditional Computation](https://mlsanity.com/paper/1308.3432/) (2013)
- [The Concrete Distribution: A Continuous Relaxation of Discrete Random Variables](https://mlsanity.com/paper/1611.00712/) (2016)
- [Importance Weighted Autoencoders](https://mlsanity.com/paper/1509.00519/) (2015)
- [Black Box Variational Inference](https://mlsanity.com/paper/1401.0118/) (2013)
- [Advances in Variational Inference](https://mlsanity.com/paper/1711.05597/) (2017)
- [Don't Blame the ELBO! A Linear VAE Perspective on Posterior Collapse](https://mlsanity.com/paper/1911.02469/) (2019)
- [Sticking the Landing: Simple, Lower-Variance Gradient Estimators for Variational Inference](https://mlsanity.com/paper/1703.09194/) (2017)
- [REBAR: Low-variance, unbiased gradient estimates for discrete latent variable models](https://mlsanity.com/paper/1703.07370/) (2017)
- [Tighter Variational Bounds are Not Necessarily Better](https://mlsanity.com/paper/1802.04537/) (2018)
- [Resampled Priors for Variational Autoencoders](https://mlsanity.com/paper/1810.11428/) (2018)
- [An Alternative Probabilistic Interpretation of the Huber Loss](https://mlsanity.com/paper/1911.02088/) (2019)
- [Doubly Reparameterized Gradient Estimators for Monte Carlo Objectives](https://mlsanity.com/paper/1810.04152/) (2018)
- [Variational Autoencoder with Learned Latent Structure](https://mlsanity.com/paper/2006.10597/) (2020)
- [Reinterpreting Importance-Weighted Autoencoders](https://mlsanity.com/paper/1704.02916/) (2017)
- [Rao-Blackwellizing the Straight-Through Gumbel-Softmax Gradient Estimator](https://mlsanity.com/paper/2010.04838/) (2020)
- [Pathwise Derivatives Beyond the Reparameterization Trick](https://mlsanity.com/paper/1806.01851/) (2018)
- [Variational Rejection Sampling](https://mlsanity.com/paper/1804.01712/) (2018)
- [A Structured Variational Auto-encoder for Learning Deep Hierarchies of Sparse Features](https://mlsanity.com/paper/1602.08734/) (2016)
- [Evolutionary Variational Optimization of Generative Models](https://mlsanity.com/paper/2012.12294/) (2020)

## Built on by (most cited first, 6 of 6)

- [Decomposed Linear Dynamical Systems (dLDS) for learning the latent components of neural dynamics](https://mlsanity.com/paper/2206.02972/) (2022)
- [SC-VAE: Sparse Coding-based Variational Autoencoder with Learned ISTA](https://mlsanity.com/paper/2303.16666/) (2023)
- [Manifold Contrastive Learning with Variational Lie Group Operators](https://mlsanity.com/paper/2306.13544/) (2023)
- [Learning Sparse Codes with Entropy-Based ELBOs](https://mlsanity.com/paper/2311.01888/) (2023)
- [Factor Analysis with Correlated Topic Model for Multi-Modal Data](https://mlsanity.com/paper/2504.18914/) (2025)
- [Variational Sparse Paired Autoencoders (vsPAIR) for Inverse Problems and Uncertainty Quantification](https://mlsanity.com/paper/2602.02948/) (2026)

## Abstract (arXiv)

Sparse coding strategies have been lauded for their parsimonious representations of data that leverage low dimensional structure. However, inference of these codes typically relies on an optimization procedure with poor computational scaling in high-dimensional problems. For example, sparse inference in the representations learned in the high-dimensional intermediary layers of deep neural networks (DNNs) requires an iterative minimization to be performed at each training step. As such, recent, quick methods in variational inference have been proposed to infer sparse codes by learning a distribution over the codes with a DNN. In this work, we propose a new approach to variational sparse coding that allows us to learn sparse distributions by thresholding samples, avoiding the use of problematic relaxations. We first evaluate and analyze our method by training a linear generator, showing that it has superior performance, statistical efficiency, and gradient estimation compared to other sparse distributions. We then compare to a standard variational autoencoder using a DNN generator on the Fashion MNIST and CelebA datasets