# Variational Pretraining for Semi-supervised Text Classification

arXiv: https://arxiv.org/abs/1906.02242 · HTML: https://mlsanity.com/paper/1906.02242/

## Key facts

- Cited by 41 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 5 Jun 2019.

## Built on by (most cited first, 30 of 41)

- [AMMUS : A Survey of Transformer-based Pretrained Models in Natural Language Processing](https://mlsanity.com/paper/2108.05542/) (2021)
- [Picking on the Same Person: Does Algorithmic Monoculture lead to Outcome Homogenization?](https://mlsanity.com/paper/2211.13972/) (2022)
- [Latent Diffusion Energy-Based Model for Interpretable Text Modeling](https://mlsanity.com/paper/2206.05895/) (2022)
- [Variational Information Bottleneck for Effective Low-Resource Fine-Tuning](https://mlsanity.com/paper/2106.05469/) (2021)
- [AMMU : A Survey of Transformer-based Biomedical Pretrained Language Models](https://mlsanity.com/paper/2105.00827/) (2021)
- [Get more for less: Principled Data Selection for Warming Up Fine-Tuning in LLMs](https://mlsanity.com/paper/2405.02774/) (2024)
- [The Rich Get Richer: Disparate Impact of Semi-Supervised Learning](https://mlsanity.com/paper/2110.06282/) (2021)
- [TELEClass: Taxonomy Enrichment and LLM-Enhanced Hierarchical Text Classification with Minimal Supervision](https://mlsanity.com/paper/2403.00165/) (2024)
- [Don't Stop Pretraining? Make Prompt-based Fine-tuning Powerful Learner](https://mlsanity.com/paper/2305.01711/) (2023)
- [Fuse It More Deeply! A Variational Transformer with Layer-Wise Latent Variable Inference for Text Generation](https://mlsanity.com/paper/2207.06130/) (2022)
- [Rethinking Semi-supervised Learning with Language Models](https://mlsanity.com/paper/2305.13002/) (2023)
- [Weakly-Supervised Hierarchical Models for Predicting Persuasive Strategies in Good-faith Textual Requests](https://mlsanity.com/paper/2101.06351/) (2021)
- [Cross-SEAN: A Cross-Stitch Semi-Supervised Neural Attention Model for COVID-19 Fake News Detection](https://mlsanity.com/paper/2102.08924/) (2021)
- [Latent Space Energy-Based Model of Symbol-Vector Coupling for Text Generation and Classification](https://mlsanity.com/paper/2108.11556/) (2021)
- [Neighborhood-Regularized Self-Training for Learning with Few Labels](https://mlsanity.com/paper/2301.03726/) (2023)
- [Goldilocks: Just-Right Tuning of BERT for Technology-Assisted Review](https://mlsanity.com/paper/2105.01044/) (2021)
- [On Sample Based Explanation Methods for NLP:Efficiency, Faithfulness, and Semantic Evaluation](https://mlsanity.com/paper/2106.04753/) (2021)
- [Automatic Document Selection for Efficient Encoder Pretraining](https://mlsanity.com/paper/2210.10951/) (2022)
- [AdaVAE: Exploring Adaptive GPT-2s in Variational Auto-Encoders for Language Modeling](https://mlsanity.com/paper/2205.05862/) (2022)
- [Semi-Supervised Text Classification via Self-Pretraining](https://mlsanity.com/paper/2109.15300/) (2021)
- [Semi-Supervised Reward Modeling via Iterative Self-Training](https://mlsanity.com/paper/2409.06903/) (2024)
- [Self-Training with Differentiable Teacher](https://mlsanity.com/paper/2109.07049/) (2021)
- [Combining exogenous and endogenous signals with a semi-supervised co-attention network for early detection of COVID-19 fake tweets](https://mlsanity.com/paper/2104.05321/) (2021)
- [PseudoReasoner: Leveraging Pseudo Labels for Commonsense Knowledge Base Population](https://mlsanity.com/paper/2210.07988/) (2022)
- [DisCo: Distilled Student Models Co-training for Semi-supervised Text Mining](https://mlsanity.com/paper/2305.12074/) (2023)
- [ZeroVO: Visual Odometry with Minimal Assumptions](https://mlsanity.com/paper/2506.08005/) (2025)
- [Uncertainty-aware Self-training for Low-resource Neural Sequence Labeling](https://mlsanity.com/paper/2302.08659/) (2023)
- [Decouple knowledge from parameters for plug-and-play language modeling](https://mlsanity.com/paper/2305.11564/) (2023)
- [Rank-Aware Negative Training for Semi-Supervised Text Classification](https://mlsanity.com/paper/2306.07621/) (2023)
- [Uncertainty-aware Parameter-Efficient Self-training for Semi-supervised Language Understanding](https://mlsanity.com/paper/2310.13022/) (2023)

## Abstract (arXiv)

We introduce VAMPIRE, a lightweight pretraining framework for effective text classification when data and computing resources are limited. We pretrain a unigram document model as a variational autoencoder on in-domain, unlabeled data and use its internal states as features in a downstream classifier. Empirically, we show the relative strength of VAMPIRE against computationally expensive contextual embeddings and other popular semi-supervised baselines under low resource settings. We also find that fine-tuning to in-domain data is crucial to achieving decent performance from contextual embeddings when working with limited supervision. We accompany this paper with code to pretrain and use VAMPIRE embeddings in downstream tasks.