# Learning with Pseudo-Ensembles

arXiv: https://arxiv.org/abs/1412.4864 · HTML: https://mlsanity.com/paper/1412.4864/

## Key facts

- Cited by 241 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 16 Dec 2014.

## Built on by (most cited first, 30 of 241)

- [FlexMatch: Boosting Semi-Supervised Learning with Curriculum Pseudo Labeling](https://mlsanity.com/paper/2110.08263/) (2021)
- [End-to-End Semi-Supervised Object Detection with Soft Teacher](https://mlsanity.com/paper/2106.09018/) (2021)
- [FreeMatch: Self-adaptive Thresholding for Semi-supervised Learning](https://mlsanity.com/paper/2205.07246/) (2022)
- [Semi-Supervised Semantic Segmentation Using Unreliable Pseudo-Labels](https://mlsanity.com/paper/2203.03884/) (2022)
- [ST++: Make Self-training Work Better for Semi-supervised Semantic Segmentation](https://mlsanity.com/paper/2106.05095/) (2021)
- [Self-supervised Augmentation Consistency for Adapting Semantic Segmentation](https://mlsanity.com/paper/2105.00097/) (2021)
- [Dash: Semi-Supervised Learning with Dynamic Thresholding](https://mlsanity.com/paper/2109.00650/) (2021)
- [Cycle Self-Training for Domain Adaptation](https://mlsanity.com/paper/2103.03571/) (2021)
- [Instant-Teaching: An End-to-End Semi-Supervised Object Detection Framework](https://mlsanity.com/paper/2103.11402/) (2021)
- [An Empirical Survey of Data Augmentation for Limited Data Learning in NLP](https://mlsanity.com/paper/2106.07499/) (2021)
- [Prompt Consistency for Zero-Shot Task Generalization](https://mlsanity.com/paper/2205.00049/) (2022)
- [AutoEval Done Right: Using Synthetic Data for Model Evaluation](https://mlsanity.com/paper/2403.07008/) (2024)
- [BoostMIS: Boosting Medical Image Semi-supervised Learning with Adaptive Pseudo Labeling and Informative Active Annotation](https://mlsanity.com/paper/2203.02533/) (2022)
- [Antipodes of Label Differential Privacy: PATE and ALIBI](https://mlsanity.com/paper/2106.03408/) (2021)
- [Semi-Supervised and Unsupervised Deep Visual Learning: A Survey](https://mlsanity.com/paper/2208.11296/) (2022)
- [Consistency Regularization for Variational Auto-Encoders](https://mlsanity.com/paper/2105.14859/) (2021)
- [Semi-Supervised Action Recognition with Temporal Contrastive Learning](https://mlsanity.com/paper/2102.02751/) (2021)
- [Dense Teacher: Dense Pseudo-Labels for Semi-supervised Object Detection](https://mlsanity.com/paper/2207.02541/) (2022)
- [Rethinking Pseudo Labels for Semi-Supervised Object Detection](https://mlsanity.com/paper/2106.00168/) (2021)
- [SemiFL: Semi-Supervised Federated Learning for Unlabeled Clients with Alternate Training](https://mlsanity.com/paper/2106.01432/) (2021)
- [CorrMatch: Label Propagation via Correlation Matching for Semi-Supervised Semantic Segmentation](https://mlsanity.com/paper/2306.04300/) (2023)
- [Rethinking Semi-Supervised Medical Image Segmentation: A Variance-Reduction Perspective](https://mlsanity.com/paper/2302.01735/) (2023)
- [Active Teacher for Semi-Supervised Object Detection](https://mlsanity.com/paper/2303.08348/) (2023)
- [CoSSL: Co-Learning of Representation and Classifier for Imbalanced Semi-Supervised Learning](https://mlsanity.com/paper/2112.04564/) (2021)
- [SemiVL: Semi-Supervised Semantic Segmentation with Vision-Language Guidance](https://mlsanity.com/paper/2311.16241/) (2023)
- [A Realistic Evaluation of Semi-Supervised Learning for Fine-Grained Classification](https://mlsanity.com/paper/2104.00679/) (2021)
- [Revisiting Consistency Regularization for Semi-Supervised Learning](https://mlsanity.com/paper/2112.05825/) (2021)
- [Sinkhorn Label Allocation: Semi-Supervised Classification via Annealed Self-Training](https://mlsanity.com/paper/2102.08622/) (2021)
- [BoxTeacher: Exploring High-Quality Pseudo Labels for Weakly Supervised Instance Segmentation](https://mlsanity.com/paper/2210.05174/) (2022)
- [SemiFed: Semi-supervised Federated Learning with Consistency and Pseudo-Labeling](https://mlsanity.com/paper/2108.09412/) (2021)

## Abstract (arXiv)

We formalize the notion of a pseudo-ensemble, a (possibly infinite) collection of child models spawned from a parent model by perturbing it according to some noise process. E.g., dropout (Hinton et. al, 2012) in a deep neural network trains a pseudo-ensemble of child subnetworks generated by randomly masking nodes in the parent network. We present a novel regularizer based on making the behavior of a pseudo-ensemble robust with respect to the noise process generating it. In the fully-supervised setting, our regularizer matches the performance of dropout. But, unlike dropout, our regularizer naturally extends to the semi-supervised setting, where it produces state-of-the-art results. We provide a case study in which we transform the Recursive Neural Tensor Network of (Socher et. al, 2013) into a pseudo-ensemble, which significantly improves its performance on a real-world sentiment analysis benchmark.