mlsanityData to 7 Oct 2026

Towards Realistic Practices In Low-Resource Natural Language Processing: The Development Set

arXiv 1909.01522 ↗First posted 4 Sep 2019cs.CLcited by 16

Not on the map. Why?

Abstract (arXiv)

Development sets are impractical to obtain for real low-resource languages, since using all available data for training is often more effective. However, development sets are widely used in research papers that purport to deal with low-resource natural language processing (NLP). Here, we aim to answer the following questions: Does using a development set for early stopping in the low-resource setting influence results as compared to a more realistic alternative, where the number of training epochs is tuned on development languages? And does it lead to overestimation or underestimation of performance? We repeat multiple experiments from recent work on neural models for low-resource NLP and compare results for models obtained by training with and without development sets. On average over languages, absolute accuracy differs by up to 1.4%. However, for some languages and tasks, differences are as big as 18.0% accuracy. Our results highlight the importance of realistic experimental setups in the publication of low-resource NLP research results.

Key facts. Cited by 16 papers in the map (data through 7 Oct 2026). First posted on arXiv 4 Sep 2019.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 8; 2022: 3; 2023: 4; 2024: 0; 2025: 0; 2026: 1.

16 papers.

Sort:

By area:

  1. 2021AmericasNLI: Evaluating Zero-shot Natural Language Understanding of Pretrained Multilingual Models in Truly Low-resource Languagescited by 50
  2. 2021Discrete and Soft Prompting for Multilingual Modelscited by 49
  3. 2021STraTA: Self-Training with Task Augmentation for Better Few-shot Learningcited by 33
  4. 2021Benchmarking Multimodal AutoML for Tabular Data with Text Fieldscited by 24
  5. 2022Pre-Trained Multilingual Sequence-to-Sequence Models: A Hope for Low-Resource Language Translation?cited by 24
  6. 2023Cultural Compass: Predicting Transfer Learning Success in Offensive Language Detection with Cultural Featurescited by 18
  7. 2023SwitchPrompt: Learning Domain-Specific Gated Soft Prompts for Classification in Low-Resource Domainscited by 12
  8. 2021A Massively Multilingual Analysis of Cross-linguality in Shared Embedding Spacecited by 8
  9. 2021Investigating the Effect of Natural Language Explanations on Out-of-Distribution Generalization in Few-shot NLIcited by 7
  10. 2022Por Qu'e N~ao Utiliser Alla Spraak? Mixed Training with Gradient Optimization in Few-Shot Cross-Lingual Transfercited by 6
  11. 2023Interpreting Indirect Answers to Yes-No Questions in Multiple Languagescited by 2
  12. 2021AcTune: Uncertainty-aware Active Self-Training for Semi-Supervised Active Learning with Pretrained Language Modelscited by 1
4 more
  1. 2022Cross-lingual Approaches for the Detection of Adverse Drug Reactions in German from a Patient's Perspectivecited by 1
  2. 2023Not Enough Labeled Data? Just Add Semantics: A Data-Efficient Method for Inferring Online Health Textscited by 1
  3. 2021Exploiting All Samples in Low-Resource Sentence Classification: Early Stopping and Initialization Parameterscited by 0
  4. 2026Neural Induction of Finite-State Transducerscited by 0