# Semi-Supervised Active Learning with Temporal Output Discrepancy

arXiv: https://arxiv.org/abs/2107.14153 · HTML: https://mlsanity.com/paper/2107.14153/

## Key facts

- Cited by 50 papers in the map (data through 14 Oct 2026).
- Builds on 31 of its 62 references that are in the map.
- First posted on arXiv 29 Jul 2021.

## Builds on (most cited first, 30 of 31)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Intriguing properties of neural networks](https://mlsanity.com/paper/1312.6199/) (2013)
- [Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learning](https://mlsanity.com/paper/1506.02142/) (2015)
- [The Cityscapes Dataset for Semantic Urban Scene Understanding](https://mlsanity.com/paper/1604.01685/) (2016)
- [Understanding Black-box Predictions via Influence Functions](https://mlsanity.com/paper/1703.04730/) (2017)
- [Unsupervised Visual Representation Learning by Context Prediction](https://mlsanity.com/paper/1505.05192/) (2015)
- [Temporal Ensembling for Semi-Supervised Learning](https://mlsanity.com/paper/1610.02242/) (2016)
- [Virtual Adversarial Training: A Regularization Method for Supervised and Semi-Supervised Learning](https://mlsanity.com/paper/1704.03976/) (2017)
- [Deep Bayesian Active Learning with Image Data](https://mlsanity.com/paper/1703.02910/) (2017)
- [Semi-Supervised Learning with Deep Generative Models](https://mlsanity.com/paper/1406.5298/) (2014)
- [Mean teachers are better role models: Weight-averaged consistency targets improve semi-supervised deep learning results](https://mlsanity.com/paper/1703.01780/) (2017)
- [Lipschitz regularity of deep neural networks: analysis and efficient estimation](https://mlsanity.com/paper/1805.10965/) (2018)
- [Learning Loss for Active Learning](https://mlsanity.com/paper/1905.03677/) (2019)
- [Dilated Residual Networks](https://mlsanity.com/paper/1705.09914/) (2017)
- [Variational Adversarial Active Learning](https://mlsanity.com/paper/1904.00370/) (2019)
- [Cost-Effective Active Learning for Deep Image Classification](https://mlsanity.com/paper/1701.03551/) (2017)
- [Adversarial Active Learning for Deep Networks: a Margin Based Approach](https://mlsanity.com/paper/1802.09841/) (2018)
- [Suggestive Annotation: A Deep Active Learning Framework for Biomedical Image Segmentation](https://mlsanity.com/paper/1706.04737/) (2017)
- [Uncertainty-guided Continual Learning with Bayesian Neural Networks](https://mlsanity.com/paper/1906.02425/) (2019)
- [Task-Aware Variational Adversarial Active Learning](https://mlsanity.com/paper/2002.04709/) (2020)
- [There Are Many Consistent Explanations of Unlabeled Data: Why You Should Average](https://mlsanity.com/paper/1806.05594/) (2018)
- [Efficient Active Learning for Image Classification and Segmentation using a Sample Selection and Conditional Generative Adversarial Network](https://mlsanity.com/paper/1806.05473/) (2018)
- [Generative Adversarial Active Learning](https://mlsanity.com/paper/1702.07956/) (2017)
- [Sequential Graph Convolutional Network for Active Learning](https://mlsanity.com/paper/2006.10219/) (2020)
- [State-Relabeling Adversarial Active Learning](https://mlsanity.com/paper/2004.04943/) (2020)
- [Adversarial Sampling for Active Learning](https://mlsanity.com/paper/1808.06671/) (2018)
- [The Local Elasticity of Neural Networks](https://mlsanity.com/paper/1910.06943/) (2019)
- [Towards Human-Machine Cooperation: Self-supervised Sample Mining for Object Detection](https://mlsanity.com/paper/1803.09867/) (2018)
- [Cost-Sensitive Active Learning for Intracranial Hemorrhage Detection](https://mlsanity.com/paper/1809.02882/) (2018)

## Built on by (most cited first, 30 of 50)

- [A Survey on Deep Active Learning: Recent Advances and New Frontiers](https://mlsanity.com/paper/2405.00334/) (2024)
- [Active Finetuning: Exploiting Annotation Budget in the Pretraining-Finetuning Paradigm](https://mlsanity.com/paper/2303.14382/) (2023)
- [A comprehensive survey on deep active learning in medical image analysis](https://mlsanity.com/paper/2310.14230/) (2023)
- [Think Twice Before Selection: Federated Evidential Active Learning for Medical Image Analysis with Domain Shifts](https://mlsanity.com/paper/2312.02567/) (2023)
- [Unsupervised Selective Labeling for More Effective Semi-Supervised Learning](https://mlsanity.com/paper/2110.03006/) (2021)
- [Making Your First Choice: To Address Cold Start Problem in Vision Active Learning](https://mlsanity.com/paper/2210.02442/) (2022)
- [Bamboo: Building Mega-Scale Vision Dataset Continually with Human-Machine Synergy](https://mlsanity.com/paper/2203.07845/) (2022)
- [Active learning for medical image segmentation with stochastic batches](https://mlsanity.com/paper/2301.07670/) (2023)
- [LabelBench: A Comprehensive Framework for Benchmarking Adaptive Label-Efficient Learning](https://mlsanity.com/paper/2306.09910/) (2023)
- [Active Learning Strategies for Weakly-supervised Object Detection](https://mlsanity.com/paper/2207.12112/) (2022)
- [Knowledge-Aware Federated Active Learning with Non-IID Data](https://mlsanity.com/paper/2211.13579/) (2022)
- [Label-Efficient Domain Generalization via Collaborative Exploration and Generalization](https://mlsanity.com/paper/2208.03644/) (2022)
- [TiDAL: Learning Training Dynamics for Active Learning](https://mlsanity.com/paper/2210.06788/) (2022)
- [TAAL: Test-time Augmentation for Active Learning in Medical Image Segmentation](https://mlsanity.com/paper/2301.06624/) (2023)
- [Best Practices in Active Learning for Semantic Segmentation](https://mlsanity.com/paper/2302.04075/) (2023)
- [Monocular 3D Object Detection with LiDAR Guided Semi Supervised Active Learning](https://mlsanity.com/paper/2307.08415/) (2023)
- [Towards Free Data Selection with General-Purpose Models](https://mlsanity.com/paper/2309.17342/) (2023)
- [Making Better Use of Unlabelled Data in Bayesian Active Learning](https://mlsanity.com/paper/2404.17249/) (2024)
- [Multi-Task Consistency for Active Learning](https://mlsanity.com/paper/2306.12398/) (2023)
- [Confidence Estimation Using Unlabeled Data](https://mlsanity.com/paper/2307.10440/) (2023)
- [ALWOD: Active Learning for Weakly-Supervised Object Detection](https://mlsanity.com/paper/2309.07914/) (2023)
- [Consistency-Based Semi-supervised Evidential Active Learning for Diagnostic Radiograph Classification](https://mlsanity.com/paper/2209.01858/) (2022)
- [Towards General and Efficient Active Learning](https://mlsanity.com/paper/2112.07963/) (2021)
- [Deep Active Learning for Multi-Label Classification of Remote Sensing Images](https://mlsanity.com/paper/2212.01165/) (2022)
- [One Class One Click: Quasi Scene-level Weakly Supervised Point Cloud Semantic Segmentation with Active Learning](https://mlsanity.com/paper/2211.12657/) (2022)
- [Active Learning via Classifier Impact and Greedy Selection for Interactive Image Retrieval](https://mlsanity.com/paper/2412.02310/) (2024)
- [Feasibility Study on Active Learning of Smart Surrogates for Scientific Simulations](https://mlsanity.com/paper/2407.07674/) (2024)
- [MoBYv2AL: Self-supervised Active Learning for Image Classification](https://mlsanity.com/paper/2301.01531/) (2023)
- [REAL: A Representative Error-Driven Approach for Active Learning](https://mlsanity.com/paper/2307.00968/) (2023)
- [Downstream-Pretext Domain Knowledge Traceback for Active Learning](https://mlsanity.com/paper/2407.14720/) (2024)

## Abstract (arXiv)

While deep learning succeeds in a wide range of tasks, it highly depends on the massive collection of annotated data which is expensive and time-consuming. To lower the cost of data annotation, active learning has been proposed to interactively query an oracle to annotate a small proportion of informative samples in an unlabeled dataset. Inspired by the fact that the samples with higher loss are usually more informative to the model than the samples with lower loss, in this paper we present a novel deep active learning approach that queries the oracle for data annotation when the unlabeled sample is believed to incorporate high loss. The core of our approach is a measurement Temporal Output Discrepancy (TOD) that estimates the sample loss by evaluating the discrepancy of outputs given by models at different optimization steps. Our theoretical investigation shows that TOD lower-bounds the accumulated sample loss thus it can be used to select informative unlabeled samples. On basis of TOD, we further develop an effective unlabeled data sampling strategy as well as an unsupervised learning criterion that enhances model performance by incorporating the unlabeled data. Due to the simplicity of TOD, our active learning approach is efficient, flexible, and task-agnostic. Extensive experimental results demonstrate that our approach achieves superior performances than the state-of-the-art active learning methods on image classification and semantic segmentation tasks.