# Few-Sample Named Entity Recognition for Security Vulnerability Reports by Fine-Tuning Pre-Trained Language Models

arXiv: https://arxiv.org/abs/2108.06590 · HTML: https://mlsanity.com/paper/2108.06590/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 24 of its 46 references that are in the map.
- First posted on arXiv 14 Aug 2021.

## Builds on (most cited first, 24 of 24)

- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks](https://mlsanity.com/paper/1703.03400/) (2017)
- [Prototypical Networks for Few-shot Learning](https://mlsanity.com/paper/1703.05175/) (2017)
- [Matching Networks for One Shot Learning](https://mlsanity.com/paper/1606.04080/) (2016)
- [Making Pre-trained Language Models Better Few-shot Learners](https://mlsanity.com/paper/2012.15723/) (2020)
- [Neural Architectures for Named Entity Recognition](https://mlsanity.com/paper/1603.01360/) (2016)
- [Pyramid Stereo Matching Network](https://mlsanity.com/paper/1803.08669/) (2018)
- [Generalizing from a Few Examples: A Survey on Few-Shot Learning](https://mlsanity.com/paper/1904.05046/) (2019)
- [Fine-Tuning Pretrained Language Models: Weight Initializations, Data Orders, and Early Stopping](https://mlsanity.com/paper/2002.06305/) (2020)
- [On the Stability of Fine-tuning BERT: Misconceptions, Explanations, and Strong Baselines](https://mlsanity.com/paper/2006.04884/) (2020)
- [Revisiting Few-sample BERT Fine-tuning](https://mlsanity.com/paper/2006.05987/) (2020)
- [Sentence Encoders on STILTs: Supplementary Training on Intermediate Labeled-data Tasks](https://mlsanity.com/paper/1811.01088/) (2018)
- [Few-shot Slot Tagging with Collapsed Dependency Transfer and Label-enhanced Task-adaptive Projection Network](https://mlsanity.com/paper/2006.05702/) (2020)
- [Few-shot classification in Named Entity Recognition Task](https://mlsanity.com/paper/1812.06158/) (2018)
- [Few-shot Text Classification with Distributional Signatures](https://mlsanity.com/paper/1908.06039/) (2019)
- [Few-Shot Named Entity Recognition: A Comprehensive Study](https://mlsanity.com/paper/2012.14978/) (2020)
- [SwellShark: A Generative Model for Biomedical Named Entity Recognition without Labeled Data](https://mlsanity.com/paper/1704.06360/) (2017)
- [A Manually-Curated Dataset of Fixes to Vulnerabilities of Open-Source Software](https://mlsanity.com/paper/1902.02595/) (2019)
- [Automatic Labeling for Entity Extraction in Cyber Security](https://mlsanity.com/paper/1308.4941/) (2013)
- [A System for Automated Open-Source Threat Intelligence Gathering and Management](https://mlsanity.com/paper/2101.07769/) (2021)
- [Simple and Effective Few-Shot Named Entity Recognition with Structured Nearest Neighbor Learning](https://mlsanity.com/paper/2010.02405/) (2020)
- [Non-Parametric Few-Shot Learning for Word Sense Disambiguation](https://mlsanity.com/paper/2104.12677/) (2021)
- [Generating Informative CVE Description From ExploitDB Posts by Extractive Summarization](https://mlsanity.com/paper/2101.01431/) (2021)
- [An Empirical Study of Android Security Bulletins in Different Vendors](https://mlsanity.com/paper/2002.09629/) (2020)

## Built on by (most cited first, 3 of 3)

- [Few-Shot Nested Named Entity Recognition](https://mlsanity.com/paper/2212.00953/) (2022)
- [Seed-Guided Fine-Grained Entity Typing in Science and Engineering Domains](https://mlsanity.com/paper/2401.13129/) (2024)
- [Cybersecurity Entity Alignment via Masked Graph Attention Networks](https://mlsanity.com/paper/2207.01434/) (2022)

## Abstract (arXiv)

Public security vulnerability reports (e.g., CVE reports) play an important role in the maintenance of computer and network systems. Security companies and administrators rely on information from these reports to prioritize tasks on developing and deploying patches to their customers. Since these reports are unstructured texts, automatic information extraction (IE) can help scale up the processing by converting the unstructured reports to structured forms, e.g., software names and versions and vulnerability types. Existing works on automated IE for security vulnerability reports often rely on a large number of labeled training samples. However, creating massive labeled training set is both expensive and time consuming. In this work, for the first time, we propose to investigate this problem where only a small number of labeled training samples are available. In particular, we investigate the performance of fine-tuning several state-of-the-art pre-trained language models on our small training dataset. The results show that with pre-trained language models and carefully tuned hyperparameters, we have reached or slightly outperformed the state-of-the-art system on this task. Consistent with previous two-step process of first fine-tuning on main category and then transfer learning to others as in [7], if otherwise following our proposed approach, the number of required labeled samples substantially decrease in both stages: 90% reduction in fine-tuning from 5758 to 576,and 88.8% reduction in transfer learning with 64 labeled samples per category. Our experiments thus demonstrate the effectiveness of few-sample learning on NER for security vulnerability report. This result opens up multiple research opportunities for few-sample learning for security vulnerability reports, which is discussed in the paper. Code: https://github.com/guanqun-yang/FewVulnerability.