# Semi-Supervised Learning via Weight-aware Distillation under Class Distribution Mismatch

arXiv: https://arxiv.org/abs/2308.11874 · HTML: https://mlsanity.com/paper/2308.11874/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- Builds on 31 of its 54 references that are in the map.
- First posted on arXiv 23 Aug 2023.

## Builds on (most cited first, 30 of 31)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [A Simple Framework for Contrastive Learning of Visual Representations](https://mlsanity.com/paper/2002.05709/) (2020)
- [Distilling the Knowledge in a Neural Network](https://mlsanity.com/paper/1503.02531/) (2015)
- [Representation Learning with Contrastive Predictive Coding](https://mlsanity.com/paper/1807.03748/) (2018)
- [Wide Residual Networks](https://mlsanity.com/paper/1605.07146/) (2016)
- [FixMatch: Simplifying Semi-Supervised Learning with Consistency and Confidence](https://mlsanity.com/paper/2001.07685/) (2020)
- [Unsupervised Feature Learning via Non-Parametric Instance-level Discrimination](https://mlsanity.com/paper/1805.01978/) (2018)
- [MixMatch: A Holistic Approach to Semi-Supervised Learning](https://mlsanity.com/paper/1905.02249/) (2019)
- [Learning deep representations by mutual information estimation and maximization](https://mlsanity.com/paper/1808.06670/) (2018)
- [Temporal Ensembling for Semi-Supervised Learning](https://mlsanity.com/paper/1610.02242/) (2016)
- [Virtual Adversarial Training: A Regularization Method for Supervised and Semi-Supervised Learning](https://mlsanity.com/paper/1704.03976/) (2017)
- [Relational Knowledge Distillation](https://mlsanity.com/paper/1904.05068/) (2019)
- [Learning Representations by Maximizing Mutual Information Across Views](https://mlsanity.com/paper/1906.00910/) (2019)
- [A Survey on Contrastive Self-supervised Learning](https://mlsanity.com/paper/2011.00362/) (2020)
- [Do Deep Nets Really Need to be Deep?](https://mlsanity.com/paper/1312.6184/) (2013)
- [Regularization With Stochastic Transformations and Perturbations for Deep Semi-Supervised Learning](https://mlsanity.com/paper/1606.04586/) (2016)
- [CSI: Novelty Detection via Contrastive Learning on Distributionally Shifted Instances](https://mlsanity.com/paper/2007.08176/) (2020)
- [Lipschitz regularity of deep neural networks: analysis and efficient estimation](https://mlsanity.com/paper/1805.10965/) (2018)
- [Decoupled Knowledge Distillation](https://mlsanity.com/paper/2203.08679/) (2022)
- [Keystroke Dynamics as Part of Lifelogging](https://mlsanity.com/paper/2010.10144/) (2020)
- [Knowledge Distillation and Student-Teacher Learning for Visual Intelligence: A Review and New Outlooks](https://mlsanity.com/paper/2004.05937/) (2020)
- [Open-World Semi-Supervised Learning](https://mlsanity.com/paper/2102.03526/) (2021)
- [Zero-Shot Knowledge Distillation in Deep Networks](https://mlsanity.com/paper/1905.08114/) (2019)
- [Class-Aware Contrastive Semi-Supervised Learning](https://mlsanity.com/paper/2203.02261/) (2022)
- [Attention-based Ensemble for Deep Metric Learning](https://mlsanity.com/paper/1804.00382/) (2018)
- [OpenMatch: Open-set Consistency Regularization for Semi-supervised Learning with Outliers](https://mlsanity.com/paper/2105.14148/) (2021)
- [Trash to Treasure: Harvesting OOD Data with Cross-Modal Matching for Open-Set Semi-Supervised Learning](https://mlsanity.com/paper/2108.05617/) (2021)
- [OpenLDN: Learning to Discover Novel Classes for Open-World Semi-Supervised Learning](https://mlsanity.com/paper/2207.02261/) (2022)
- [OpenCoS: Contrastive Semi-supervised Learning for Handling Open-set Unlabeled Data](https://mlsanity.com/paper/2107.08943/) (2021)

## Built on by (most cited first, 8 of 8)

- [Multimodal Unsupervised Domain Generalization by Retrieving Across the Modality Gap](https://mlsanity.com/paper/2402.04416/) (2024)
- [Simple yet Effective Semi-supervised Knowledge Distillation from Vision-Language Models via Dual-Head Optimization](https://mlsanity.com/paper/2505.07675/) (2025)
- [Diversify and Conquer: Open-set Disagreement for Robust Semi-supervised Learning with Outliers](https://mlsanity.com/paper/2505.24443/) (2025)
- [The Championship-Winning Solution for the 5th CLVISION Challenge 2024](https://mlsanity.com/paper/2406.16615/) (2024)
- [Open-Set Heterogeneous Domain Adaptation: Theoretical Analysis and Algorithm](https://mlsanity.com/paper/2412.13036/) (2024)
- [PCoreSet: Effective Active Learning through Knowledge Distillation from Vision-Language Models](https://mlsanity.com/paper/2506.00910/) (2025)
- [Dimensionally Reduced Open-World Clustering: DROWCULA](https://mlsanity.com/paper/2509.07184/) (2025)
- [From Skeletons to Pixels: Few-Shot Precise Event Spotting via Representation and Prediction Distillation](https://mlsanity.com/paper/2604.22839/) (2026)

## Abstract (arXiv)

Semi-Supervised Learning (SSL) under class distribution mismatch aims to tackle a challenging problem wherein unlabeled data contain lots of unknown categories unseen in the labeled ones. In such mismatch scenarios, traditional SSL suffers severe performance damage due to the harmful invasion of the instances with unknown categories into the target classifier. In this study, by strict mathematical reasoning, we reveal that the SSL error under class distribution mismatch is composed of pseudo-labeling error and invasion error, both of which jointly bound the SSL population risk. To alleviate the SSL error, we propose a robust SSL framework called Weight-Aware Distillation (WAD) that, by weights, selectively transfers knowledge beneficial to the target task from unsupervised contrastive representation to the target classifier. Specifically, WAD captures adaptive weights and high-quality pseudo labels to target instances by exploring point mutual information (PMI) in representation space to maximize the role of unlabeled data and filter unknown categories. Theoretically, we prove that WAD has a tight upper bound of population risk under class distribution mismatch. Experimentally, extensive results demonstrate that WAD outperforms five state-of-the-art SSL approaches and one standard baseline on two benchmark datasets, CIFAR10 and CIFAR100, and an artificial cross-dataset. The code is available at https://github.com/RUC-DWBI-ML/research/tree/main/WAD-master.