# Certainty Driven Consistency Loss on Multi-Teacher Networks for Semi-Supervised Learning

arXiv: https://arxiv.org/abs/1901.05657 · HTML: https://mlsanity.com/paper/1901.05657/

## Key facts

- Cited by 11 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 17 Jan 2019.

## Built on by (most cited first, 11 of 11)

- [Advancements in Point Cloud Data Augmentation for Deep Learning: A Survey](https://mlsanity.com/paper/2308.12113/) (2023)
- [Enhancing MR Image Segmentation with Realistic Adversarial Data Augmentation](https://mlsanity.com/paper/2108.03429/) (2021)
- [SLaM: Student-Label Mixing for Distillation with Unlabeled Examples](https://mlsanity.com/paper/2302.03806/) (2023)
- [Robust Active Distillation](https://mlsanity.com/paper/2210.01213/) (2022)
- [Self-Reinforcing Prototype Evolution with Dual-Knowledge Cooperation for Semi-Supervised Lifelong Person Re-Identification](https://mlsanity.com/paper/2507.01884/) (2025)
- [Cache & Distil: Optimising API Calls to Large Language Models](https://mlsanity.com/paper/2310.13561/) (2023)
- [LADDER: Latent Boundary-guided Adversarial Training](https://mlsanity.com/paper/2206.03717/) (2022)
- [Weighting Pseudo-Labels via High-Activation Feature Index Similarity and Object Detection for Semi-Supervised Segmentation](https://mlsanity.com/paper/2407.12630/) (2024)
- [Generalized Open-World Semi-Supervised Object Detection](https://mlsanity.com/paper/2307.15710/) (2023)
- [Linear Projections of Teacher Embeddings for Few-Class Distillation](https://mlsanity.com/paper/2409.20449/) (2024)
- [Dense Uncertainty Estimation via an Ensemble-based Conditional Latent Variable Model](https://mlsanity.com/paper/2111.11055/) (2021)

## Abstract (arXiv)

One of the successful approaches in semi-supervised learning is based on the consistency regularization. Typically, a student model is trained to be consistent with teacher prediction for the inputs under different perturbations. To be successful, the prediction targets given by teacher should have good quality, otherwise the student can be misled by teacher. Unfortunately, existing methods do not assess the quality of the teacher targets. In this paper, we propose a novel Certainty-driven Consistency Loss (CCL) that exploits the predictive uncertainty in the consistency loss to let the student dynamically learn from reliable targets. Specifically, we propose two approaches, i.e. Filtering CCL and Temperature CCL to either filter out uncertain predictions or pay less attention on them in the consistency regularization. We further introduce a novel decoupled framework to encourage model difference. Experimental results on SVHN, CIFAR-10, and CIFAR-100 demonstrate the advantages of our method over a few existing methods.