# Distilling Named Entity Recognition Models for Endangered Species from Large Language Models

arXiv: https://arxiv.org/abs/2403.15430 · HTML: https://mlsanity.com/paper/2403.15430/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 12 of its 19 references that are in the map.
- First posted on arXiv 13 Mar 2024.

## Builds on (most cited first, 12 of 12)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Distilling the Knowledge in a Neural Network](https://mlsanity.com/paper/1503.02531/) (2015)
- [BioBERT: a pre-trained biomedical language representation model for biomedical text mining](https://mlsanity.com/paper/1901.08746/) (2019)
- [Domain-Specific Language Model Pretraining for Biomedical Natural Language Processing](https://mlsanity.com/paper/2007.15779/) (2020)
- [Snorkel: Rapid Training Data Creation with Weak Supervision](https://mlsanity.com/paper/1711.10160/) (2017)
- [Matching the Blanks: Distributional Similarity for Relation Learning](https://mlsanity.com/paper/1906.03158/) (2019)
- [Large Language Models are Few-Shot Clinical Information Extractors](https://mlsanity.com/paper/2205.12689/) (2022)
- [Want To Reduce Labeling Cost? GPT-3 Can Help](https://mlsanity.com/paper/2108.13487/) (2021)
- [Thinking about GPT-3 In-Context Learning for Biomedical IE? Think Again](https://mlsanity.com/paper/2203.08410/) (2022)
- [Language Models in the Loop: Incorporating Prompting into Weak Supervision](https://mlsanity.com/paper/2205.02318/) (2022)
- [Improving Multi-Task Deep Neural Networks via Knowledge Distillation for Natural Language Understanding](https://mlsanity.com/paper/1904.09482/) (2019)

## Built on by (most cited first, 1 of 1)

- [MAviS: A Multimodal Conversational Assistant For Avian Species](https://mlsanity.com/paper/2603.07294/) (2026)

## Abstract (arXiv)

Natural language processing (NLP) practitioners are leveraging large language models (LLM) to create structured datasets from semi-structured and unstructured data sources such as patents, papers, and theses, without having domain-specific knowledge. At the same time, ecological experts are searching for a variety of means to preserve biodiversity. To contribute to these efforts, we focused on endangered species and through in-context learning, we distilled knowledge from GPT-4. In effect, we created datasets for both named entity recognition (NER) and relation extraction (RE) via a two-stage process: 1) we generated synthetic data from GPT-4 of four classes of endangered species, 2) humans verified the factual accuracy of the synthetic data, resulting in gold data. Eventually, our novel dataset contains a total of 3.6K sentences, evenly divided between 1.8K NER and 1.8K RE sentences. The constructed dataset was then used to fine-tune both general BERT and domain-specific BERT variants, completing the knowledge distillation process from GPT-4 to BERT, because GPT-4 is resource intensive. Experiments show that our knowledge transfer approach is effective at creating a NER model suitable for detecting endangered species from texts.