mlsanityData to 14 Oct 2026

Fine-tuning BERT for Low-Resource Natural Language Understanding via Active Learning

arXiv 2012.02462 ↗First posted 4 Dec 2020cs.CLcited by 25

Not on the map. Why?

Abstract (arXiv)

Recently, leveraging pre-trained Transformer based language models in down stream, task specific models has advanced state of the art results in natural language understanding tasks. However, only a little research has explored the suitability of this approach in low resource settings with less than 1,000 training data points. In this work, we explore fine-tuning methods of BERT -- a pre-trained Transformer based language model -- by utilizing pool-based active learning to speed up training while keeping the cost of labeling new data constant. Our experimental results on the GLUE data set show an advantage in model performance by maximizing the approximate knowledge gain of the model when querying from the pool of unlabeled data. Finally, we demonstrate and analyze the benefits of freezing layers of the language model during fine-tuning to reduce the number of trainable parameters, making it more suitable for low-resource settings.

Key facts. Cited by 25 papers in the map (data through 14 Oct 2026). First posted on arXiv 4 Dec 2020.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2021: 4; 2022: 6; 2023: 7; 2024: 5; 2025: 1; 2026: 2.

25 papers.

Sort:

  1. 2021Fine-Tuning Large Neural Language Models for Biomedical Natural Language Processingcited by 57
  2. 2022A Survey of Active Learning for Natural Language Processingcited by 40
  3. 2024ActiveLLM: Large Language Model-based Active Learning for Textual Few-Shot Scenarioscited by 18
  4. 2022Multi-task Active Learning for Pre-trained Transformer-based Modelscited by 16
  5. 2022Active Few-Shot Learning with FASLcited by 11
  6. 2022Multilingual Detection of Personal Employment Status on Twittercited by 8
  7. 2021Bayesian Active Summarizationcited by 5
  8. 2022Human-Guided Fair Classification for Natural Language Processingcited by 5
  9. 2023Towards understanding neural collapse in supervised contrastive learning with the information bottleneck methodcited by 5
  10. 2023Parameter-Efficient Language Model Tuning with Active Learning in Low-Resource Settingscited by 5
  11. 2024AnchorAL: Computationally Efficient Active Learning for Large and Imbalanced Datasetscited by 5
  12. 2021Active learning for reducing labeling effort in text classification taskscited by 4
13 more
  1. 2022Smooth Sailing: Improving Active Learning for Pre-trained Language Models with Representation Smoothness Analysiscited by 4
  2. 2023Interactive Multi-fidelity Learning for Cost-effective Adaptation of Language Model with Sparse Human Supervisioncited by 4
  3. 2024Designing Domain-Specific Large Language Models: The Critical Role of Fine-Tuning in Public Opinion Simulationcited by 4
  4. 2023On Dataset Transferability in Active Learning for Transformerscited by 3
  5. 2021Emojich -- zero-shot emoji generation using Russian language: a technical reportcited by 2
  6. 2023Teacher Intervention: Improving Convergence of Quantization Aware Training for Ultra-Low Precision Transformerscited by 2
  7. 2024Recent Advances in Data-Driven Business Process Managementcited by 2
  8. 2023An Eye on Clinical BERT: Investigating Language Model Generalization for Diabetic Eye Disease Phenotypingcited by 1
  9. 2023Cross-lingual German Biomedical Information Extraction: from Zero-shot to Human-in-the-Loopcited by 0
  10. 2024Calpric: Inclusive and Fine-grain Labeling of Privacy Policies with Crowdsourcing and Active Learningcited by 0
  11. 2025ProtoBERT-LoRA: Parameter-Efficient Prototypical Finetuning for Immunotherapy Study Identificationcited by 0
  12. 2026Beyond Subtokens: A Rich Character Embedding for Low-resource and Morphologically Complex Languagescited by 0
  13. 2026Evaluating Zero-Shot and One-Shot Adaptation of Small Language Models in Leader-Follower Interactioncited by 0