# Exploiting All Samples in Low-Resource Sentence Classification: Early Stopping and Initialization Parameters

arXiv: https://arxiv.org/abs/2111.06971 · HTML: https://mlsanity.com/paper/2111.06971/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 33 of its 76 references that are in the map.
- First posted on arXiv 12 Nov 2021.

## Builds on (most cited first, 30 of 33)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [mixup: Beyond Empirical Risk Minimization](https://mlsanity.com/paper/1710.09412/) (2017)
- [On Calibration of Modern Neural Networks](https://mlsanity.com/paper/1706.04599/) (2017)
- [Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks](https://mlsanity.com/paper/1703.03400/) (2017)
- [Reformer: The Efficient Transformer](https://mlsanity.com/paper/2001.04451/) (2020)
- [Averaging Weights Leads to Wider Optima and Better Generalization](https://mlsanity.com/paper/1803.05407/) (2018)
- [Convolutional Neural Networks for Sentence Classification](https://mlsanity.com/paper/1408.5882/) (2014)
- [End to End Learning for Self-Driving Cars](https://mlsanity.com/paper/1604.07316/) (2016)
- [EDA: Easy Data Augmentation Techniques for Boosting Performance on Text Classification Tasks](https://mlsanity.com/paper/1901.11196/) (2019)
- [An Overview of Multi-Task Learning in Deep Neural Networks](https://mlsanity.com/paper/1706.05098/) (2017)
- [Learning to Reweight Examples for Robust Deep Learning](https://mlsanity.com/paper/1803.09050/) (2018)
- [Generalizing from a Few Examples: A Survey on Few-Shot Learning](https://mlsanity.com/paper/1904.05046/) (2019)
- [Measuring Calibration in Deep Learning](https://mlsanity.com/paper/1904.01685/) (2019)
- [Supervised Learning of Universal Sentence Representations from Natural Language Inference Data](https://mlsanity.com/paper/1705.02364/) (2017)
- [A Survey of Data Augmentation Approaches for NLP](https://mlsanity.com/paper/2105.03075/) (2021)
- [On Mixup Training: Improved Calibration and Predictive Uncertainty for Deep Neural Networks](https://mlsanity.com/paper/1905.11001/) (2019)
- [SentEval: An Evaluation Toolkit for Universal Sentence Representations](https://mlsanity.com/paper/1803.05449/) (2018)
- [Not Enough Data? Deep Learning to the Rescue!](https://mlsanity.com/paper/1911.03118/) (2019)
- [A Survey on Recent Approaches for Natural Language Processing in Low-Resource Scenarios](https://mlsanity.com/paper/2010.12309/) (2020)
- [Transfer Learning and Distant Supervision for Multilingual Transformer Models: A Study on African Languages](https://mlsanity.com/paper/2010.03179/) (2020)
- [Learning Data Manipulation for Augmentation and Weighting](https://mlsanity.com/paper/1910.12795/) (2019)
- [Early Stopping without a Validation Set](https://mlsanity.com/paper/1703.09580/) (2017)
- [Low Resource Text Classification with ULMFit and Backtranslation](https://mlsanity.com/paper/1903.09244/) (2019)
- [RATT: Leveraging Unlabeled Data to Guarantee Generalization](https://mlsanity.com/paper/2105.00303/) (2021)
- [Privacy in Social Media: Identification, Mitigation and Applications](https://mlsanity.com/paper/1808.02191/) (2018)
- [Towards Realistic Practices In Low-Resource Natural Language Processing: The Development Set](https://mlsanity.com/paper/1909.01522/) (2019)
- [SPlit: An Optimal Method for Data Splitting](https://mlsanity.com/paper/2012.10945/) (2020)
- [Establishing Baselines for Text Classification in Low-Resource Languages](https://mlsanity.com/paper/2005.02068/) (2020)
- [Channel-Wise Early Stopping without a Validation Set via NNK Polytope Interpolation](https://mlsanity.com/paper/2107.12972/) (2021)

## Abstract (arXiv)

To improve deep-learning performance in low-resource settings, many researchers have redesigned model architectures or applied additional data (e.g., external resources, unlabeled samples). However, there have been relatively few discussions on how to make good use of small amounts of labeled samples, although it is potentially beneficial and should be done before applying additional data or redesigning models. In this study, we assume a low-resource setting in which only a few labeled samples (i.e., 30-100 per class) are available, and we discuss how to exploit them without additional data or model redesigns. We explore possible approaches in the following three aspects: training-validation splitting, early stopping, and weight initialization. Extensive experiments are conducted on six public sentence classification datasets. Performance on various evaluation metrics (e.g., accuracy, loss, and calibration error) significantly varied depending on the approaches that were combined in the three aspects. Based on the results, we propose an integrated method, which is to initialize the model with a weight averaging method and use a non-validation stop method to train all samples. This simple integrated method consistently outperforms the competitive methods; e.g., the average accuracy of six datasets of this method was 1.8% higher than those of conventional validation-based methods. In addition, the integrated method further improves the performance when adapted to several state-of-the-art models that use additional data or redesign the network architecture (e.g., self-training and enhanced structural models). Our results highlight the importance of the training strategy and suggest that the integrated method can be the first step in the low-resource setting. This study provides empirical knowledge that will be helpful when dealing with low-resource data in future efforts.