# Neighborhood-Regularized Self-Training for Learning with Few Labels

arXiv: https://arxiv.org/abs/2301.03726 · HTML: https://mlsanity.com/paper/2301.03726/

## Key facts

- Cited by 14 papers in the map (data through 14 Oct 2026).
- Builds on 51 of its 74 references that are in the map.
- First posted on arXiv 10 Jan 2023.

## Builds on (most cited first, 30 of 51)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [A Simple Framework for Contrastive Learning of Visual Representations](https://mlsanity.com/paper/2002.05709/) (2020)
- [On Calibration of Modern Neural Networks](https://mlsanity.com/paper/1706.04599/) (2017)
- [Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learning](https://mlsanity.com/paper/1506.02142/) (2015)
- [Billion-scale similarity search with GPUs](https://mlsanity.com/paper/1702.08734/) (2017)
- [Character-level Convolutional Networks for Text Classification](https://mlsanity.com/paper/1509.01626/) (2015)
- [HuggingFace's Transformers: State-of-the-art Natural Language Processing](https://mlsanity.com/paper/1910.03771/) (2019)
- [FixMatch: Simplifying Semi-Supervised Learning with Consistency and Confidence](https://mlsanity.com/paper/2001.07685/) (2020)
- [BioBERT: a pre-trained biomedical language representation model for biomedical text mining](https://mlsanity.com/paper/1901.08746/) (2019)
- [Self-training with Noisy Student improves ImageNet classification](https://mlsanity.com/paper/1911.04252/) (2019)
- [Unsupervised Data Augmentation for Consistency Training](https://mlsanity.com/paper/1904.12848/) (2019)
- [MoleculeNet: A Benchmark for Molecular Machine Learning](https://mlsanity.com/paper/1703.00564/) (2017)
- [Strategies for Pre-training Graph Neural Networks](https://mlsanity.com/paper/1905.12265/) (2019)
- [Virtual Adversarial Training: A Regularization Method for Supervised and Semi-Supervised Learning](https://mlsanity.com/paper/1704.03976/) (2017)
- [Long-tail learning via logit adjustment](https://mlsanity.com/paper/2007.07314/) (2020)
- [FlexMatch: Boosting Semi-Supervised Learning with Curriculum Pseudo Labeling](https://mlsanity.com/paper/2110.08263/) (2021)
- [Graph Contrastive Learning with Adaptive Augmentation](https://mlsanity.com/paper/2010.14945/) (2020)
- [Pseudo-Labeling and Confirmation Bias in Deep Semi-Supervised Learning](https://mlsanity.com/paper/1908.02983/) (2019)
- [InfoGraph: Unsupervised and Semi-supervised Graph-Level Representation Learning via Mutual Information Maximization](https://mlsanity.com/paper/1908.01000/) (2019)
- [In Defense of Pseudo-Labeling: An Uncertainty-Aware Pseudo-label Selection Framework for Semi-Supervised Learning](https://mlsanity.com/paper/2101.06329/) (2021)
- [Graph Contrastive Learning Automated](https://mlsanity.com/paper/2106.07594/) (2021)
- [A Survey on Deep Semi-supervised Learning](https://mlsanity.com/paper/2103.00550/) (2021)
- [MixText: Linguistically-Informed Interpolation of Hidden Space for Semi-Supervised Text Classification](https://mlsanity.com/paper/2004.12239/) (2020)
- [CoMatch: Semi-supervised Learning with Contrastive Graph Regularization](https://mlsanity.com/paper/2011.11183/) (2020)
- [Revisiting Self-Training for Neural Sequence Generation](https://mlsanity.com/paper/1909.13788/) (2019)
- [SimMatch: Semi-supervised Learning with Similarity Matching](https://mlsanity.com/paper/2203.06915/) (2022)
- [Self-training Improves Pre-training for Natural Language Understanding](https://mlsanity.com/paper/2010.02194/) (2020)
- [WRENCH: A Comprehensive Benchmark for Weak Supervision](https://mlsanity.com/paper/2109.11377/) (2021)

## Built on by (most cited first, 14 of 14)

- [AutoReP: Automatic ReLU Replacement for Fast Private Network Inference](https://mlsanity.com/paper/2308.10134/) (2023)
- [LinGCN: Structural Linearized Graph Convolutional Network for Homomorphically Encrypted Inference](https://mlsanity.com/paper/2309.14331/) (2023)
- [ReGen: Zero-Shot Text Classification via Training Data Generation with Progressive Dense Retrieval](https://mlsanity.com/paper/2305.10703/) (2023)
- [PTGB: Pre-Train Graph Neural Networks for Brain Network Analysis](https://mlsanity.com/paper/2305.14376/) (2023)
- [DyGen: Learning from Noisy Labels via Dynamics-Enhanced Generative Modeling](https://mlsanity.com/paper/2305.19395/) (2023)
- [R-Mixup: Riemannian Mixup for Biological Networks](https://mlsanity.com/paper/2306.02532/) (2023)
- [Weakly-Supervised Scientific Document Classification via Retrieval-Augmented Multi-Stage Training](https://mlsanity.com/paper/2306.07193/) (2023)
- [How Many Validation Labels Do You Need? Exploring the Design Space of Label-Efficient Model Ranking](https://mlsanity.com/paper/2312.01619/) (2023)
- [QuMoS: A Framework for Preserving Security of Quantum Machine Learning Model](https://mlsanity.com/paper/2304.11511/) (2023)
- [Open Visual Knowledge Extraction via Relation-Oriented Multimodality Model Prompting](https://mlsanity.com/paper/2310.18804/) (2023)
- [PV2TEA: Patching Visual Modality to Textual-Established Information Extraction](https://mlsanity.com/paper/2306.01016/) (2023)
- [CoLaDa: A Collaborative Label Denoising Framework for Cross-lingual Named Entity Recognition](https://mlsanity.com/paper/2305.14913/) (2023)
- [CAST: Cluster-Aware Self-Training for Tabular Data via Reliable Confidence](https://mlsanity.com/paper/2310.06380/) (2023)
- [Self-Training for Sample-Efficient Active Learning for Text Classification with Pre-Trained Language Models](https://mlsanity.com/paper/2406.09206/) (2024)

## Abstract (arXiv)

Training deep neural networks (DNNs) with limited supervision has been a popular research topic as it can significantly alleviate the annotation burden. Self-training has been successfully applied in semi-supervised learning tasks, but one drawback of self-training is that it is vulnerable to the label noise from incorrect pseudo labels. Inspired by the fact that samples with similar labels tend to share similar representations, we develop a neighborhood-based sample selection approach to tackle the issue of noisy pseudo labels. We further stabilize self-training via aggregating the predictions from different rounds during sample selection. Experiments on eight tasks show that our proposed method outperforms the strongest self-training baseline with 1.83% and 2.51% performance gain for text and graph datasets on average. Our further analysis demonstrates that our proposed data selection strategy reduces the noise of pseudo labels by 36.8% and saves 57.3% of the time when compared with the best baseline. Our code and appendices will be uploaded to https://github.com/ritaranx/NeST.