# Towards Realistic Practices In Low-Resource Natural Language Processing: The Development Set

arXiv: https://arxiv.org/abs/1909.01522 · HTML: https://mlsanity.com/paper/1909.01522/

## Key facts

- Cited by 16 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 4 Sep 2019.

## Built on by (most cited first, 16 of 16)

- [AmericasNLI: Evaluating Zero-shot Natural Language Understanding of Pretrained Multilingual Models in Truly Low-resource Languages](https://mlsanity.com/paper/2104.08726/) (2021)
- [Discrete and Soft Prompting for Multilingual Models](https://mlsanity.com/paper/2109.03630/) (2021)
- [STraTA: Self-Training with Task Augmentation for Better Few-shot Learning](https://mlsanity.com/paper/2109.06270/) (2021)
- [Benchmarking Multimodal AutoML for Tabular Data with Text Fields](https://mlsanity.com/paper/2111.02705/) (2021)
- [Pre-Trained Multilingual Sequence-to-Sequence Models: A Hope for Low-Resource Language Translation?](https://mlsanity.com/paper/2203.08850/) (2022)
- [Cultural Compass: Predicting Transfer Learning Success in Offensive Language Detection with Cultural Features](https://mlsanity.com/paper/2310.06458/) (2023)
- [SwitchPrompt: Learning Domain-Specific Gated Soft Prompts for Classification in Low-Resource Domains](https://mlsanity.com/paper/2302.06868/) (2023)
- [A Massively Multilingual Analysis of Cross-linguality in Shared Embedding Space](https://mlsanity.com/paper/2109.06324/) (2021)
- [Investigating the Effect of Natural Language Explanations on Out-of-Distribution Generalization in Few-shot NLI](https://mlsanity.com/paper/2110.06223/) (2021)
- [Por Qu'e N~ao Utiliser Alla Spraak? Mixed Training with Gradient Optimization in Few-Shot Cross-Lingual Transfer](https://mlsanity.com/paper/2204.13869/) (2022)
- [Interpreting Indirect Answers to Yes-No Questions in Multiple Languages](https://mlsanity.com/paper/2310.13290/) (2023)
- [AcTune: Uncertainty-aware Active Self-Training for Semi-Supervised Active Learning with Pretrained Language Models](https://mlsanity.com/paper/2112.08787/) (2021)
- [Cross-lingual Approaches for the Detection of Adverse Drug Reactions in German from a Patient's Perspective](https://mlsanity.com/paper/2208.02031/) (2022)
- [Not Enough Labeled Data? Just Add Semantics: A Data-Efficient Method for Inferring Online Health Texts](https://mlsanity.com/paper/2309.09877/) (2023)
- [Exploiting All Samples in Low-Resource Sentence Classification: Early Stopping and Initialization Parameters](https://mlsanity.com/paper/2111.06971/) (2021)
- [Neural Induction of Finite-State Transducers](https://mlsanity.com/paper/2601.10918/) (2026)

## Abstract (arXiv)

Development sets are impractical to obtain for real low-resource languages, since using all available data for training is often more effective. However, development sets are widely used in research papers that purport to deal with low-resource natural language processing (NLP). Here, we aim to answer the following questions: Does using a development set for early stopping in the low-resource setting influence results as compared to a more realistic alternative, where the number of training epochs is tuned on development languages? And does it lead to overestimation or underestimation of performance? We repeat multiple experiments from recent work on neural models for low-resource NLP and compare results for models obtained by training with and without development sets. On average over languages, absolute accuracy differs by up to 1.4%. However, for some languages and tasks, differences are as big as 18.0% accuracy. Our results highlight the importance of realistic experimental setups in the publication of low-resource NLP research results.