mlsanityData to 7 Oct 2026

Variational Pretraining for Semi-supervised Text Classification

arXiv 1906.02242 ↗First posted 5 Jun 2019cs.CL cs.LGcited by 41

Not on the map. Why?

Abstract (arXiv)

We introduce VAMPIRE, a lightweight pretraining framework for effective text classification when data and computing resources are limited. We pretrain a unigram document model as a variational autoencoder on in-domain, unlabeled data and use its internal states as features in a downstream classifier. Empirically, we show the relative strength of VAMPIRE against computationally expensive contextual embeddings and other popular semi-supervised baselines under low resource settings. We also find that fine-tuning to in-domain data is crucial to achieving decent performance from contextual embeddings when working with limited supervision. We accompany this paper with code to pretrain and use VAMPIRE embeddings in downstream tasks.

Key facts. Cited by 41 papers in the map (data through 7 Oct 2026). First posted on arXiv 5 Jun 2019.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 15; 2022: 10; 2023: 11; 2024: 3; 2025: 1; 2026: 1.

41 papers.

Sort:

By area:

  1. 2021AMMUS : A Survey of Transformer-based Pretrained Models in Natural Language Processingcited by 131
  2. 2022Picking on the Same Person: Does Algorithmic Monoculture lead to Outcome Homogenization?cited by 89
  3. 2022Latent Diffusion Energy-Based Model for Interpretable Text Modelingcited by 67
  4. 2021Variational Information Bottleneck for Effective Low-Resource Fine-Tuningcited by 51
  5. 2021AMMU : A Survey of Transformer-based Biomedical Pretrained Language Modelscited by 32
  6. 2024Get more for less: Principled Data Selection for Warming Up Fine-Tuning in LLMscited by 32
  7. 2021The Rich Get Richer: Disparate Impact of Semi-Supervised Learningcited by 28
  8. 2024TELEClass: Taxonomy Enrichment and LLM-Enhanced Hierarchical Text Classification with Minimal Supervisioncited by 25
  9. 2023Don't Stop Pretraining? Make Prompt-based Fine-tuning Powerful Learnercited by 23
  10. 2022Fuse It More Deeply! A Variational Transformer with Layer-Wise Latent Variable Inference for Text Generationcited by 20
  11. 2023Rethinking Semi-supervised Learning with Language Modelscited by 17
  12. 2021Weakly-Supervised Hierarchical Models for Predicting Persuasive Strategies in Good-faith Textual Requestscited by 16
29 more
  1. 2021Cross-SEAN: A Cross-Stitch Semi-Supervised Neural Attention Model for COVID-19 Fake News Detectioncited by 15
  2. 2021Latent Space Energy-Based Model of Symbol-Vector Coupling for Text Generation and Classificationcited by 14
  3. 2023Neighborhood-Regularized Self-Training for Learning with Few Labelscited by 14
  4. 2021On Sample Based Explanation Methods for NLP:Efficiency, Faithfulness, and Semantic Evaluationcited by 12
  5. 2021Goldilocks: Just-Right Tuning of BERT for Technology-Assisted Reviewcited by 11
  6. 2022Automatic Document Selection for Efficient Encoder Pretrainingcited by 11
  7. 2022AdaVAE: Exploring Adaptive GPT-2s in Variational Auto-Encoders for Language Modelingcited by 10
  8. 2021Semi-Supervised Text Classification via Self-Pretrainingcited by 8
  9. 2024Semi-Supervised Reward Modeling via Iterative Self-Trainingcited by 8
  10. 2021Self-Training with Differentiable Teachercited by 7
  11. 2021Combining exogenous and endogenous signals with a semi-supervised co-attention network for early detection of COVID-19 fake tweetscited by 5
  12. 2022PseudoReasoner: Leveraging Pseudo Labels for Commonsense Knowledge Base Populationcited by 5
  13. 2023DisCo: Distilled Student Models Co-training for Semi-supervised Text Miningcited by 4
  14. 2025ZeroVO: Visual Odometry with Minimal Assumptionscited by 4
  15. 2023Uncertainty-aware Self-training for Low-resource Neural Sequence Labelingcited by 3
  16. 2023Decouple knowledge from parameters for plug-and-play language modelingcited by 3
  17. 2023Rank-Aware Negative Training for Semi-Supervised Text Classificationcited by 3
  18. 2023Uncertainty-aware Parameter-Efficient Self-training for Semi-supervised Language Understandingcited by 3
  19. 2022Exploiting Unlabeled Data for Target-Oriented Opinion Words Extractioncited by 2
  20. 2022Time Will Change Things: An Empirical Study on Dynamic Language Understanding in Social Media Classificationcited by 2
  21. 2023Neural Networks Against (and For) Self-Training: Classification with Small Labeled and Large Unlabeled Setscited by 2
  22. 2022Are We Really Making Much Progress in Text Classification? A Comparative Reviewcited by 1
  23. 2021The Sensitivity of Word Embeddings-based Author Detection Models to Semantic-preserving Adversarial Perturbationscited by 0
  24. 2021Challenging the Semi-Supervised VAE Framework for Text Classificationcited by 0
  25. 2021Entropy optimized semi-supervised decomposed vector-quantized variational autoencoder model based on transfer learning for multiclass text classification and generationcited by 0
  26. 2022Transformer-Based Conditioned Variational Autoencoder for Dialogue Generationcited by 0
  27. 2023AutoWS: Automated Weak Supervision Framework for Text Classificationcited by 0
  28. 2023Substituting Data Annotation with Balanced Updates and Collective Loss in Multi-label Text Classificationcited by 0
  29. 2026Semi-Supervised Learning with Balanced Deep Representation Distributionscited by 0