# DiZiNER: Disagreement-guided Instruction Refinement via Pilot Annotation Simulation for Zero-shot Named Entity Recognition

arXiv: https://arxiv.org/abs/2604.15866 · HTML: https://mlsanity.com/paper/2604.15866/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 14 of its 38 references that are in the map.
- First posted on arXiv 17 Apr 2026.

## Builds on (most cited first, 14 of 14)

- [Unified Structure Generation for Universal Information Extraction](https://mlsanity.com/paper/2203.12277/) (2022)
- [InstructUIE: Multi-task Instruction Tuning for Unified Information Extraction](https://mlsanity.com/paper/2304.08085/) (2023)
- [GoLLIE: Annotation Guidelines improve Zero-Shot Information-Extraction](https://mlsanity.com/paper/2310.03668/) (2023)
- [CrossNER: Evaluating Cross-Domain Named Entity Recognition](https://mlsanity.com/paper/2012.04373/) (2020)
- [CodeIE: Large Code Generation Models are Better Few-Shot Information Extractors](https://mlsanity.com/paper/2305.05711/) (2023)
- [Empirical Study of Zero-Shot NER with ChatGPT](https://mlsanity.com/paper/2310.10035/) (2023)
- [NuNER: Entity Recognition Encoder Pre-training via LLM-Annotated Data](https://mlsanity.com/paper/2402.15343/) (2024)
- [KnowCoder: Coding Structured Knowledge into LLMs for Universal Information Extraction](https://mlsanity.com/paper/2403.07969/) (2024)
- [Cross-type Biomedical Named Entity Recognition with Deep Multi-Task Learning](https://mlsanity.com/paper/1801.09851/) (2018)
- [ADELIE: Aligning Large Language Models on Information Extraction](https://mlsanity.com/paper/2405.05008/) (2024)
- [Rethinking Negative Instances for Generative Named Entity Recognition](https://mlsanity.com/paper/2402.16602/) (2024)
- [Few-shot clinical entity recognition in English, French and Spanish: masked language models outperform generative model prompting](https://mlsanity.com/paper/2402.12801/) (2024)
- [Re-Examine Distantly Supervised NER: A New Benchmark and a Simple Approach](https://mlsanity.com/paper/2402.14948/) (2024)
- [Annotation Errors and NER: A Study with OntoNotes 5.0](https://mlsanity.com/paper/2406.19172/) (2024)

## Abstract (arXiv)

Large language models (LLMs) have advanced information extraction (IE) by enabling zero-shot and few-shot named entity recognition (NER), yet their generative outputs still show persistent and systematic errors. Despite progress through instruction fine-tuning, zero-shot NER still lags far behind supervised systems. These recurring errors mirror inconsistencies observed in early-stage human annotation processes that resolve disagreements through pilot annotation. Motivated by this analogy, we introduce DiZiNER (Disagreement-guided Instruction Refinement via Pilot Annotation Simulation for Zero-shot Named Entity Recognition), a framework that simulates the pilot annotation process, employing LLMs to act as both annotators and supervisors. Multiple heterogeneous LLMs annotate shared texts, and a supervisor model analyzes inter-model disagreements to refine task instructions. Across 18 benchmarks, DiZiNER achieves zero-shot SOTA results on 14 datasets, improving prior bests by +8.0 F1 and reducing the zero-shot to supervised gap by over +11 points. It also consistently outperforms its supervisor, GPT-5 mini, indicating that improvements stem from disagreement-guided instruction refinement rather than model capacity. Pairwise agreement between models shows a strong correlation with NER performance, further supporting this finding.