# Variational Autoencoder for Deep Learning of Images, Labels and Captions

arXiv: https://arxiv.org/abs/1609.08976 · HTML: https://mlsanity.com/paper/1609.08976/

## Key facts

- Cited by 137 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 28 Sep 2016.

## Built on by (most cited first, 30 of 137)

- [UniVLA: Learning to Act Anywhere with Task-centric Latent Actions](https://mlsanity.com/paper/2505.06111/) (2025)
- [In Defense of Pseudo-Labeling: An Uncertainty-Aware Pseudo-label Selection Framework for Semi-Supervised Learning](https://mlsanity.com/paper/2101.06329/) (2021)
- [Spatial-Phase Shallow Learning: Rethinking Face Forgery Detection in Frequency Domain](https://mlsanity.com/paper/2103.01856/) (2021)
- [Graph neural networks for materials science and chemistry](https://mlsanity.com/paper/2208.09481/) (2022)
- [Variational Mixture-of-Experts Autoencoders for Multi-Modal Deep Generative Models](https://mlsanity.com/paper/1911.03393/) (2019)
- [Efficient Self-supervised Vision Transformers for Representation Learning](https://mlsanity.com/paper/2106.09785/) (2021)
- [UNIT-DDPM: UNpaired Image Translation with Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2104.05358/) (2021)
- [HNeRV: A Hybrid Neural Representation for Videos](https://mlsanity.com/paper/2304.02633/) (2023)
- [Explainability of deep vision-based autonomous driving systems: Review and challenges](https://mlsanity.com/paper/2101.05307/) (2021)
- [RETRIEVE: Coreset Selection for Efficient and Robust Semi-Supervised Learning](https://mlsanity.com/paper/2106.07760/) (2021)
- [Advances in 3D Generation: A Survey](https://mlsanity.com/paper/2401.17807/) (2024)
- [Root-aligned SMILES: A Tight Representation for Chemical Reaction Prediction](https://mlsanity.com/paper/2203.11444/) (2022)
- [OpenLDN: Learning to Discover Novel Classes for Open-World Semi-Supervised Learning](https://mlsanity.com/paper/2207.02261/) (2022)
- [Dynamic-Group-Aware Networks for Multi-Agent Trajectory Prediction with Relational Reasoning](https://mlsanity.com/paper/2206.13114/) (2022)
- [Towards Realistic Semi-Supervised Learning](https://mlsanity.com/paper/2207.02269/) (2022)
- [Factor Analysis, Probabilistic Principal Component Analysis, Variational Inference, and Variational Autoencoder: Tutorial and Survey](https://mlsanity.com/paper/2101.00734/) (2021)
- [CNeRV: Content-adaptive Neural Representation for Visual Data](https://mlsanity.com/paper/2211.10421/) (2022)
- [Scale-MIA: A Scalable Model Inversion Attack against Secure Federated Learning via Latent Space Reconstruction](https://mlsanity.com/paper/2311.05808/) (2023)
- [TrAISformer -- A Transformer Network with Sparse Augmented Data Representation and Cross Entropy Loss for AIS-based Vessel Trajectory Prediction](https://mlsanity.com/paper/2109.03958/) (2021)
- [Latent Denoising Makes Good Tokenizers](https://mlsanity.com/paper/2507.15856/) (2025)
- [MirrorCheck: Efficient Adversarial Defense for Vision-Language Models](https://mlsanity.com/paper/2406.09250/) (2024)
- [TimeBalance: Temporally-Invariant and Temporally-Distinctive Video Representations for Semi-Supervised Action Recognition](https://mlsanity.com/paper/2303.16268/) (2023)
- [Attention-based Convolutional Autoencoders for 3D-Variational Data Assimilation](https://mlsanity.com/paper/2101.02121/) (2021)
- [Simulating Fluids in Real-World Still Images](https://mlsanity.com/paper/2204.11335/) (2022)
- [A survey and taxonomy of loss functions in machine learning](https://mlsanity.com/paper/2301.05579/) (2023)
- [Unsupervised Learning for Human Sensing Using Radio Signals](https://mlsanity.com/paper/2207.02370/) (2022)
- [Flow Matching Meets Biology and Life Science: A Survey](https://mlsanity.com/paper/2507.17731/) (2025)
- [Distinguished Quantized Guidance for Diffusion-based Sequence Recommendation](https://mlsanity.com/paper/2501.17670/) (2025)
- [Machine learning for modelling unstructured grid data in computational physics: a review](https://mlsanity.com/paper/2502.09346/) (2025)
- [SIGMAN:Scaling 3D Human Gaussian Generation with Millions of Assets](https://mlsanity.com/paper/2504.06982/) (2025)

## Abstract (arXiv)

A novel variational autoencoder is developed to model images, as well as associated labels or captions. The Deep Generative Deconvolutional Network (DGDN) is used as a decoder of the latent image features, and a deep Convolutional Neural Network (CNN) is used as an image encoder; the CNN is used to approximate a distribution for the latent DGDN features/code. The latent code is also linked to generative models for labels (Bayesian support vector machine) or captions (recurrent neural network). When predicting a label/caption for a new image at test, averaging is performed across the distribution of latent codes; this is computationally efficient as a consequence of the learned CNN-based encoder. Since the framework is capable of modeling the image in the presence/absence of associated labels/captions, a new semi-supervised setting is manifested for CNN learning with images; the framework even allows unsupervised CNN learning, based on images alone.