# Scoping Review of Active Learning Strategies and their Evaluation Environments for Entity Recognition Tasks

arXiv: https://arxiv.org/abs/2407.03895 · HTML: https://mlsanity.com/paper/2407.03895/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 25 of its 115 references that are in the map.
- First posted on arXiv 4 Jul 2024.

## Builds on (most cited first, 25 of 25)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Cross-lingual Language Model Pretraining](https://mlsanity.com/paper/1901.07291/) (2019)
- [A Survey of Deep Active Learning](https://mlsanity.com/paper/2009.00236/) (2020)
- [A Survey of Data Augmentation Approaches for NLP](https://mlsanity.com/paper/2105.03075/) (2021)
- [Deep Active Learning for Named Entity Recognition](https://mlsanity.com/paper/1707.05928/) (2017)
- [Deep Bayesian Active Learning for Natural Language Processing: Results of a Large-Scale Empirical Study](https://mlsanity.com/paper/1808.05697/) (2018)
- [A Comparative Survey of Deep Active Learning](https://mlsanity.com/paper/2203.13450/) (2022)
- [Red teaming ChatGPT via Jailbreaking: Bias, Robustness, Reliability and Toxicity](https://mlsanity.com/paper/2301.12867/) (2023)
- [On Statistical Bias In Active Learning: How and When To Fix It](https://mlsanity.com/paper/2101.11665/) (2021)
- [OpenTag: Open Attribute Value Extraction from Product Profiles [Deep Learning, Active Learning, Named Entity Recognition]](https://mlsanity.com/paper/1806.01264/) (2018)
- [A Survey of Active Learning for Natural Language Processing](https://mlsanity.com/paper/2210.10109/) (2022)
- [Active Learning for Sequence Tagging with Deep Pre-trained Models and Bayesian Uncertainty Estimates](https://mlsanity.com/paper/2101.08133/) (2021)
- [Fair Active Learning](https://mlsanity.com/paper/2001.01796/) (2020)
- [skweak: Weak Supervision Made Easy for NLP](https://mlsanity.com/paper/2104.09683/) (2021)
- [A Survey on Cost Types, Interaction Schemes, and Annotator Performance Models in Selection Algorithms for Active Learning in Classification](https://mlsanity.com/paper/2109.11301/) (2021)
- [D-CALM: A Dynamic Clustering-based Active Learning Approach for Mitigating Bias](https://mlsanity.com/paper/2305.17013/) (2023)
- [Large Language Models are legal but they are not: Making the case for a powerful LegalLLM](https://mlsanity.com/paper/2311.08890/) (2023)
- [A Little Annotation does a Lot of Good: A Study in Bootstrapping Low-resource Named Entity Recognizers](https://mlsanity.com/paper/1908.08983/) (2019)
- [FAMIE: A Fast Active Learning Framework for Multilingual Information Extraction](https://mlsanity.com/paper/2202.08316/) (2022)
- [Jointly Learning Span Extraction and Sequence Labeling for Information Extraction from Business Documents](https://mlsanity.com/paper/2205.13434/) (2022)
- [On Efficiently Acquiring Annotations for Multilingual Models](https://mlsanity.com/paper/2204.01016/) (2022)
- [ALE: A Simulation-Based Active Learning Evaluation Framework for the Parameter-Driven Comparison of Query Strategies for NLP](https://mlsanity.com/paper/2308.02537/) (2023)
- [Focusing on Potential Named Entities During Active Label Acquisition](https://mlsanity.com/paper/2111.03837/) (2021)

## Built on by (most cited first, 2 of 2)

- [Reassessing Active Learning Adoption in Contemporary NLP: A Community Survey](https://mlsanity.com/paper/2503.09701/) (2025)
- [Active Learning Methods for Efficient Data Utilization and Model Performance Enhancement](https://mlsanity.com/paper/2504.16136/) (2025)

## Abstract (arXiv)

We conducted a scoping review for active learning in the domain of natural language processing (NLP), which we summarize in accordance with the PRISMA-ScR guidelines as follows: Objective: Identify active learning strategies that were proposed for entity recognition and their evaluation environments (datasets, metrics, hardware, execution time). Design: We used Scopus and ACM as our search engines. We compared the results with two literature surveys to assess the search quality. We included peer-reviewed English publications introducing or comparing active learning strategies for entity recognition. Results: We analyzed 62 relevant papers and identified 106 active learning strategies. We grouped them into three categories: exploitation-based (60x), exploration-based (14x), and hybrid strategies (32x). We found that all studies used the F1-score as an evaluation metric. Information about hardware (6x) and execution time (13x) was only occasionally included. The 62 papers used 57 different datasets to evaluate their respective strategies. Most datasets contained newspaper articles or biomedical/medical data. Our analysis revealed that 26 out of 57 datasets are publicly accessible. Conclusion: Numerous active learning strategies have been identified, along with significant open questions that still need to be addressed. Researchers and practitioners face difficulties when making data-driven decisions about which active learning strategy to adopt. Conducting comprehensive empirical comparisons using the evaluation environment proposed in this study could help establish best practices in the domain.