# NuNER: Entity Recognition Encoder Pre-training via LLM-Annotated Data

arXiv: https://arxiv.org/abs/2402.15343 · HTML: https://mlsanity.com/paper/2402.15343/

## Key facts

- Cited by 34 papers in the map (data through 14 Oct 2026).
- Builds on 26 of its 37 references that are in the map.
- First posted on arXiv 23 Feb 2024.

## Builds on (most cited first, 26 of 26)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [A Simple Framework for Contrastive Learning of Visual Representations](https://mlsanity.com/paper/2002.05709/) (2020)
- [Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena](https://mlsanity.com/paper/2306.05685/) (2023)
- [UMAP: Uniform Manifold Approximation and Projection for Dimension Reduction](https://mlsanity.com/paper/1802.03426/) (2018)
- [DeBERTa: Decoding-enhanced BERT with Disentangled Attention](https://mlsanity.com/paper/2006.03654/) (2020)
- [BioBERT: a pre-trained biomedical language representation model for biomedical text mining](https://mlsanity.com/paper/1901.08746/) (2019)
- [SciBERT: A Pretrained Language Model for Scientific Text](https://mlsanity.com/paper/1903.10676/) (2019)
- [ChatGPT Outperforms Crowd-Workers for Text-Annotation Tasks](https://mlsanity.com/paper/2303.15056/) (2023)
- [InstructUIE: Multi-task Instruction Tuning for Unified Information Extraction](https://mlsanity.com/paper/2304.08085/) (2023)
- [GPT-NER: Named Entity Recognition via Large Language Models](https://mlsanity.com/paper/2304.10428/) (2023)
- [Few-NERD: A Few-Shot Named Entity Recognition Dataset](https://mlsanity.com/paper/2105.07464/) (2021)
- [GoLLIE: Annotation Guidelines improve Zero-Shot Information-Extraction](https://mlsanity.com/paper/2310.03668/) (2023)
- [GLiNER: Generalist Model for Named Entity Recognition using Bidirectional Transformer](https://mlsanity.com/paper/2311.08526/) (2023)
- [GPTEval: A Survey on Assessments of ChatGPT and GPT-4](https://mlsanity.com/paper/2308.12488/) (2023)
- [UniversalNER: Targeted Distillation from Large Language Models for Open Named Entity Recognition](https://mlsanity.com/paper/2308.03279/) (2023)
- [Low-Resource Name Tagging Learned with Weakly Labeled Data](https://mlsanity.com/paper/1908.09659/) (2019)
- [Coarse-to-Fine Pre-training for Named Entity Recognition](https://mlsanity.com/paper/2010.08210/) (2020)
- [Optimizing Bi-Encoder for Named Entity Recognition via Contrastive Learning](https://mlsanity.com/paper/2208.14565/) (2022)
- [NER-BERT: A Pre-trained Model for Low-Resource Entity Tagging](https://mlsanity.com/paper/2112.00405/) (2021)
- [Type-Aware Decomposed Framework for Few-Shot Named Entity Recognition](https://mlsanity.com/paper/2302.06397/) (2023)

## Built on by (most cited first, 30 of 34)

- [GLiREL -- Generalist Model for Zero-Shot Relation Extraction](https://mlsanity.com/paper/2501.03172/) (2025)
- [Overview of BioASQ 2025: The Thirteenth BioASQ Challenge on Large-Scale Biomedical Semantic Indexing and Question Answering](https://mlsanity.com/paper/2508.20554/) (2025)
- [GLiNER2: An Efficient Multi-Task Information Extraction System with Schema-Driven Interface](https://mlsanity.com/paper/2507.18546/) (2025)
- [ReflectDiffu:Reflect between Emotion-intent Contagion and Mimicry for Empathetic Response Generation via a RL-Diffusion Framework](https://mlsanity.com/paper/2409.10289/) (2024)
- [HistoLens: An LLM-Powered Framework for Multi-Layered Analysis of Historical Texts -- A Case Application of Yantie Lun](https://mlsanity.com/paper/2411.09978/) (2024)
- [WhisperNER: Unified Open Named Entity and Speech Recognition](https://mlsanity.com/paper/2409.08107/) (2024)
- [GLiNER-BioMed: A Suite of Efficient Models for Open Biomedical Named Entity Recognition](https://mlsanity.com/paper/2504.00676/) (2025)
- [FiNERweb: Datasets and Artifacts for Scalable Multilingual Named Entity Recognition](https://mlsanity.com/paper/2512.13884/) (2025)
- [Automatic Classification of News Subjects in Broadcast News: Application to a Gender Bias Representation Analysis](https://mlsanity.com/paper/2407.14180/) (2024)
- [SubRegWeigh: Effective and Efficient Annotation Weighing with Subword Regularization](https://mlsanity.com/paper/2409.06216/) (2024)
- [Retrieval-Enhanced Named Entity Recognition](https://mlsanity.com/paper/2410.13118/) (2024)
- [KG2QA: Knowledge Graph-enhanced Retrieval-augmented Generation for Communication Standards Question Answering](https://mlsanity.com/paper/2506.07037/) (2025)
- [SpeechLLM: Unified Speech and Language Model for Enhanced Multi-Task Understanding in Low Resource Settings](https://mlsanity.com/paper/2509.04473/) (2025)
- [MultiSoc-4D: A Benchmark for Diagnosing Instruction-Induced Label Collapse in Closed-Set LLM Annotation of Bengali Social Media](https://mlsanity.com/paper/2605.06940/) (2026)
- [GLiNER-Relex: A Unified Framework for Joint Named Entity Recognition and Relation Extraction](https://mlsanity.com/paper/2605.10108/) (2026)
- [LELA: An End-to-end LLM-based Entity Linking Framework with Zero-shot Domain Adaptation](https://mlsanity.com/paper/2605.26956/) (2026)
- [Does Synthetic Data Help Named Entity Recognition for Low-Resource Languages?](https://mlsanity.com/paper/2505.16814/) (2025)
- [Arce: Augmented Roberta with Contextualized Elucidations for Ner in Automated Rule Checking](https://mlsanity.com/paper/2508.07286/) (2025)
- [A Tale of LLMs and Induced Small Proxies: Scalable Small Language Models for Knowledge Mining](https://mlsanity.com/paper/2510.01427/) (2025)
- [Named Entity Recognition in COVID-19 tweets with Entity Knowledge Augmentation](https://mlsanity.com/paper/2510.04001/) (2025)
- [Multi-Task Pre-Finetuning of Lightweight Transformer Encoders for Text Classification and NER](https://mlsanity.com/paper/2510.07566/) (2025)
- [What Matters When Building Universal Multilingual Named Entity Recognition Models?](https://mlsanity.com/paper/2601.06347/) (2026)
- [A Semi-Automated Annotation Workflow for Paediatric Histopathology Reports Using Small Language Models](https://mlsanity.com/paper/2604.04168/) (2026)
- [Just Pass Twice: Efficient Token Classification with LLMs for Zero-Shot NER](https://mlsanity.com/paper/2604.05158/) (2026)
- [Universal NER v2: Towards a Massively Multilingual Named Entity Recognition Benchmark](https://mlsanity.com/paper/2604.12744/) (2026)
- [DiZiNER: Disagreement-guided Instruction Refinement via Pilot Annotation Simulation for Zero-shot Named Entity Recognition](https://mlsanity.com/paper/2604.15866/) (2026)
- [Decoding Text Spans for Efficient and Accurate Named-Entity Recognition](https://mlsanity.com/paper/2604.20447/) (2026)
- [Opir: Efficient Multi-Task Safety Classification for Toxicity, Jailbreaks, Hate Speech, and Harmful Content](https://mlsanity.com/paper/2605.29659/) (2026)
- [Scaling Performance and Low-Resource Annotation with Many-Shot In-Context Learning for Named Entity Recognition](https://mlsanity.com/paper/2606.21890/) (2026)
- [Task Decomposition for Efficient Annotation](https://mlsanity.com/paper/2606.24734/) (2026)

## Abstract (arXiv)

Large Language Models (LLMs) have shown impressive abilities in data annotation, opening the way for new approaches to solve classic NLP problems. In this paper, we show how to use LLMs to create NuNER, a compact language representation model specialized in the Named Entity Recognition (NER) task. NuNER can be fine-tuned to solve downstream NER problems in a data-efficient way, outperforming similar-sized foundation models in the few-shot regime and competing with much larger LLMs. We find that the size and entity-type diversity of the pre-training dataset are key to achieving good performance. We view NuNER as a member of the broader family of task-specific foundation models, recently unlocked by LLMs.