# SC-VAE: Sparse Coding-based Variational Autoencoder with Learned ISTA

arXiv: https://arxiv.org/abs/2303.16666 · HTML: https://mlsanity.com/paper/2303.16666/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 34 of its 65 references that are in the map.
- First posted on arXiv 29 Mar 2023.

## Builds on (most cited first, 30 of 34)

- [The Unreasonable Effectiveness of Deep Features as a Perceptual Metric](https://mlsanity.com/paper/1801.03924/) (2018)
- [A Style-Based Generator Architecture for Generative Adversarial Networks](https://mlsanity.com/paper/1812.04948/) (2018)
- [Neural Discrete Representation Learning](https://mlsanity.com/paper/1711.00937/) (2017)
- [Taming Transformers for High-Resolution Image Synthesis](https://mlsanity.com/paper/2012.09841/) (2020)
- [Perceptual Losses for Real-Time Style Transfer and Super-Resolution](https://mlsanity.com/paper/1603.08155/) (2016)
- [Estimating or Propagating Gradients Through Stochastic Neurons for Conditional Computation](https://mlsanity.com/paper/1308.3432/) (2013)
- [Group Normalization](https://mlsanity.com/paper/1803.08494/) (2018)
- [Object-Centric Learning with Slot Attention](https://mlsanity.com/paper/2006.15055/) (2020)
- [Autoregressive Image Generation using Residual Quantization](https://mlsanity.com/paper/2203.01941/) (2022)
- [Vector-quantized Image Modeling with Improved VQGAN](https://mlsanity.com/paper/2110.04627/) (2021)
- [Disentangling by Factorising](https://mlsanity.com/paper/1802.05983/) (2018)
- [Jukebox: A Generative Model for Music](https://mlsanity.com/paper/2005.00341/) (2020)
- [Autoencoding beyond pixels using a learned similarity metric](https://mlsanity.com/paper/1512.09300/) (2015)
- [Multi-Object Representation Learning with Iterative Variational Inference](https://mlsanity.com/paper/1903.00450/) (2019)
- [Variational Inference of Disentangled Latent Concepts from Unlabeled Observations](https://mlsanity.com/paper/1711.00848/) (2017)
- [Skin Lesion Analysis toward Melanoma Detection: A Challenge at the International Symposium on Biomedical Imaging (ISBI) 2016, hosted by the International Skin Imaging Collaboration (ISIC)](https://mlsanity.com/paper/1605.01397/) (2016)
- [MoVQ: Modulating Quantized Vectors for High-Fidelity Image Generation](https://mlsanity.com/paper/2209.09002/) (2022)
- [Preventing Posterior Collapse with delta-VAEs](https://mlsanity.com/paper/1901.03416/) (2019)
- [Variational Autoencoders Pursue PCA Directions (by Accident)](https://mlsanity.com/paper/1812.06775/) (2018)
- [Unsupervised Object Segmentation by Redrawing](https://mlsanity.com/paper/1905.13539/) (2019)
- [Image Reconstruction: From Sparsity to Data-adaptive Methods and Machine Learning](https://mlsanity.com/paper/1904.02816/) (2019)
- [W-Net: A Deep Model for Fully Unsupervised Image Segmentation](https://mlsanity.com/paper/1711.08506/) (2017)
- [Emergence of Object Segmentation in Perturbed Generative Models](https://mlsanity.com/paper/1905.12663/) (2019)
- [Multi-VAE: Learning Disentangled View-common and View-peculiar Visual Representations for Multi-view Clustering](https://mlsanity.com/paper/2106.11232/) (2021)
- [On Implicit Regularization in β-VAEs](https://mlsanity.com/paper/2002.00041/) (2020)
- [Data Efficient Unsupervised Domain Adaptation for Cross-Modality Image Segmentation](https://mlsanity.com/paper/1907.02766/) (2019)
- [OneGAN: Simultaneous Unsupervised Learning of Conditional Image Generation, Foreground Segmentation, and Fine-Grained Clustering](https://mlsanity.com/paper/1912.13471/) (2019)
- [Unsupervised Foreground Extraction via Deep Region Competition](https://mlsanity.com/paper/2110.15497/) (2021)
- [Weakly Supervised Disentanglement by Pairwise Similarities](https://mlsanity.com/paper/1906.01044/) (2019)
- [Information-Theoretic Segmentation by Inpainting Error Maximization](https://mlsanity.com/paper/2012.07287/) (2020)

## Built on by (most cited first, 4 of 4)

- [Poisson Variational Autoencoder](https://mlsanity.com/paper/2405.14473/) (2024)
- [SurvReLU: Inherently Interpretable Survival Analysis via Deep ReLU Networks](https://mlsanity.com/paper/2407.14463/) (2024)
- [A Unifying Framework for Unsupervised Concept Extraction](https://mlsanity.com/paper/2604.24936/) (2026)
- [Automatic Neuronal Activity Segmentation in Fast Four Dimensional Spatio-Temporal Fluorescence Imaging using Bayesian Approach](https://mlsanity.com/paper/2512.19032/) (2025)

## Abstract (arXiv)

Learning rich data representations from unlabeled data is a key challenge towards applying deep learning algorithms in downstream tasks. Several variants of variational autoencoders (VAEs) have been proposed to learn compact data representations by encoding high-dimensional data in a lower dimensional space. Two main classes of VAEs methods may be distinguished depending on the characteristics of the meta-priors that are enforced in the representation learning step. The first class of methods derives a continuous encoding by assuming a static prior distribution in the latent space. The second class of methods learns instead a discrete latent representation using vector quantization (VQ) along with a codebook. However, both classes of methods suffer from certain challenges, which may lead to suboptimal image reconstruction results. The first class suffers from posterior collapse, whereas the second class suffers from codebook collapse. To address these challenges, we introduce a new VAE variant, termed sparse coding-based VAE with learned ISTA (SC-VAE), which integrates sparse coding within variational autoencoder framework. The proposed method learns sparse data representations that consist of a linear combination of a small number of predetermined orthogonal atoms. The sparse coding problem is solved using a learnable version of the iterative shrinkage thresholding algorithm (ISTA). Experiments on two image datasets demonstrate that our model achieves improved image reconstruction results compared to state-of-the-art methods. Moreover, we demonstrate that the use of learned sparse code vectors allows us to perform downstream tasks like image generation and unsupervised image segmentation through clustering image patches.