# Fine-tuning BERT for Low-Resource Natural Language Understanding via Active Learning

arXiv: https://arxiv.org/abs/2012.02462 · HTML: https://mlsanity.com/paper/2012.02462/

## Key facts

- Cited by 25 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 4 Dec 2020.

## Built on by (most cited first, 25 of 25)

- [Fine-Tuning Large Neural Language Models for Biomedical Natural Language Processing](https://mlsanity.com/paper/2112.07869/) (2021)
- [A Survey of Active Learning for Natural Language Processing](https://mlsanity.com/paper/2210.10109/) (2022)
- [ActiveLLM: Large Language Model-based Active Learning for Textual Few-Shot Scenarios](https://mlsanity.com/paper/2405.10808/) (2024)
- [Multi-task Active Learning for Pre-trained Transformer-based Models](https://mlsanity.com/paper/2208.05379/) (2022)
- [Active Few-Shot Learning with FASL](https://mlsanity.com/paper/2204.09347/) (2022)
- [Multilingual Detection of Personal Employment Status on Twitter](https://mlsanity.com/paper/2203.09178/) (2022)
- [Bayesian Active Summarization](https://mlsanity.com/paper/2110.04480/) (2021)
- [Human-Guided Fair Classification for Natural Language Processing](https://mlsanity.com/paper/2212.10154/) (2022)
- [Towards understanding neural collapse in supervised contrastive learning with the information bottleneck method](https://mlsanity.com/paper/2305.11957/) (2023)
- [Parameter-Efficient Language Model Tuning with Active Learning in Low-Resource Settings](https://mlsanity.com/paper/2305.14576/) (2023)
- [AnchorAL: Computationally Efficient Active Learning for Large and Imbalanced Datasets](https://mlsanity.com/paper/2404.05623/) (2024)
- [Active learning for reducing labeling effort in text classification tasks](https://mlsanity.com/paper/2109.04847/) (2021)
- [Smooth Sailing: Improving Active Learning for Pre-trained Language Models with Representation Smoothness Analysis](https://mlsanity.com/paper/2212.11680/) (2022)
- [Interactive Multi-fidelity Learning for Cost-effective Adaptation of Language Model with Sparse Human Supervision](https://mlsanity.com/paper/2310.20153/) (2023)
- [Designing Domain-Specific Large Language Models: The Critical Role of Fine-Tuning in Public Opinion Simulation](https://mlsanity.com/paper/2409.19308/) (2024)
- [On Dataset Transferability in Active Learning for Transformers](https://mlsanity.com/paper/2305.09807/) (2023)
- [Emojich -- zero-shot emoji generation using Russian language: a technical report](https://mlsanity.com/paper/2112.02448/) (2021)
- [Teacher Intervention: Improving Convergence of Quantization Aware Training for Ultra-Low Precision Transformers](https://mlsanity.com/paper/2302.11812/) (2023)
- [Recent Advances in Data-Driven Business Process Management](https://mlsanity.com/paper/2406.01786/) (2024)
- [An Eye on Clinical BERT: Investigating Language Model Generalization for Diabetic Eye Disease Phenotyping](https://mlsanity.com/paper/2311.08687/) (2023)
- [Cross-lingual German Biomedical Information Extraction: from Zero-shot to Human-in-the-Loop](https://mlsanity.com/paper/2301.09908/) (2023)
- [Calpric: Inclusive and Fine-grain Labeling of Privacy Policies with Crowdsourcing and Active Learning](https://mlsanity.com/paper/2401.08038/) (2024)
- [ProtoBERT-LoRA: Parameter-Efficient Prototypical Finetuning for Immunotherapy Study Identification](https://mlsanity.com/paper/2503.20179/) (2025)
- [Beyond Subtokens: A Rich Character Embedding for Low-resource and Morphologically Complex Languages](https://mlsanity.com/paper/2602.21377/) (2026)
- [Evaluating Zero-Shot and One-Shot Adaptation of Small Language Models in Leader-Follower Interaction](https://mlsanity.com/paper/2602.23312/) (2026)

## Abstract (arXiv)

Recently, leveraging pre-trained Transformer based language models in down stream, task specific models has advanced state of the art results in natural language understanding tasks. However, only a little research has explored the suitability of this approach in low resource settings with less than 1,000 training data points. In this work, we explore fine-tuning methods of BERT -- a pre-trained Transformer based language model -- by utilizing pool-based active learning to speed up training while keeping the cost of labeling new data constant. Our experimental results on the GLUE data set show an advantage in model performance by maximizing the approximate knowledge gain of the model when querying from the pool of unlabeled data. Finally, we demonstrate and analyze the benefits of freezing layers of the language model during fine-tuning to reduce the number of trainable parameters, making it more suitable for low-resource settings.