# Empirical Study of Zero-Shot NER with ChatGPT

arXiv: https://arxiv.org/abs/2310.10035 · HTML: https://mlsanity.com/paper/2310.10035/

## Key facts

- Cited by 60 papers in the map (data through 14 Oct 2026).
- Builds on 24 of its 37 references that are in the map.
- First posted on arXiv 16 Oct 2023.

## Builds on (most cited first, 24 of 24)

- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://mlsanity.com/paper/2307.09288/) (2023)
- [Chain-of-Thought Prompting Elicits Reasoning in Large Language Models](https://mlsanity.com/paper/2201.11903/) (2022)
- [Self-Consistency Improves Chain of Thought Reasoning in Language Models](https://mlsanity.com/paper/2203.11171/) (2022)
- [PaLM: Scaling Language Modeling with Pathways](https://mlsanity.com/paper/2204.02311/) (2022)
- [Large Language Models are Zero-Shot Reasoners](https://mlsanity.com/paper/2205.11916/) (2022)
- [Least-to-Most Prompting Enables Complex Reasoning in Large Language Models](https://mlsanity.com/paper/2205.10625/) (2022)
- [LaMDA: Language Models for Dialog Applications](https://mlsanity.com/paper/2201.08239/) (2022)
- [Automatic Chain of Thought Prompting in Large Language Models](https://mlsanity.com/paper/2210.03493/) (2022)
- [Is ChatGPT a General-Purpose Natural Language Processing Task Solver?](https://mlsanity.com/paper/2302.06476/) (2023)
- [Matching the Blanks: Distributional Similarity for Relation Learning](https://mlsanity.com/paper/1906.03158/) (2019)
- [ChatIE: Zero-Shot Information Extraction via Chatting with ChatGPT](https://mlsanity.com/paper/2302.10205/) (2023)
- [Towards Understanding Chain-of-Thought Prompting: An Empirical Study of What Matters](https://mlsanity.com/paper/2212.10001/) (2022)
- [Dice Loss for Data-imbalanced NLP Tasks](https://mlsanity.com/paper/1911.02855/) (2019)
- [GPT-NER: Named Entity Recognition via Large Language Models](https://mlsanity.com/paper/2304.10428/) (2023)
- [Named Entity Recognition as Dependency Parsing](https://mlsanity.com/paper/2005.07150/) (2020)
- [Evaluating ChatGPT's Information Extraction Capabilities: An Assessment of Performance, Explainability, Calibration, and Faithfulness](https://mlsanity.com/paper/2304.11633/) (2023)
- [A Systematic Study and Comprehensive Evaluation of ChatGPT on Benchmark Datasets](https://mlsanity.com/paper/2305.18486/) (2023)
- [Improving Named Entity Recognition by External Context Retrieving and Cooperative Learning](https://mlsanity.com/paper/2105.03654/) (2021)
- [GPT-RE: In-context Learning for Relation Extraction using Large Language Models](https://mlsanity.com/paper/2305.02105/) (2023)
- [An Empirical Study on Information Extraction using Large Language Models](https://mlsanity.com/paper/2305.14450/) (2023)
- [The Stem Cell Hypothesis: Dilemma behind Multi-Task Learning with Transformer Encoders](https://mlsanity.com/paper/2109.06939/) (2021)
- [Prompting Large Language Models with Chain-of-Thought for Few-Shot Knowledge Base Question Generation](https://mlsanity.com/paper/2310.08395/) (2023)
- [Global Pointer: Novel Efficient Span-based Approach for Named Entity Recognition](https://mlsanity.com/paper/2208.03054/) (2022)

## Built on by (most cited first, 30 of 60)

- [Large Language Models for Generative Information Extraction: A Survey](https://mlsanity.com/paper/2312.17617/) (2023)
- [Large Language Models Meet NLP: A Survey](https://mlsanity.com/paper/2405.12819/) (2024)
- [C-ICL: Contrastive In-context Learning for Information Extraction](https://mlsanity.com/paper/2402.11254/) (2024)
- [MMREC: LLM Based Multi-Modal Recommender System](https://mlsanity.com/paper/2408.04211/) (2024)
- [A Simple but Effective Approach to Improve Structured Language Model Output for Information Extraction](https://mlsanity.com/paper/2402.13364/) (2024)
- [Self-Improving for Zero-Shot Named Entity Recognition with Large Language Models](https://mlsanity.com/paper/2311.08921/) (2023)
- [UrbanKGent: A Unified Large Language Model Agent Framework for Urban Knowledge Graph Construction](https://mlsanity.com/paper/2402.06861/) (2024)
- [Data Imputation using Large Language Model to Accelerate Recommendation System](https://mlsanity.com/paper/2407.10078/) (2024)
- [ULTRA: Unleash LLMs' Potential for Event Argument Extraction through Hierarchical Modeling and Pair-wise Self-Refinement](https://mlsanity.com/paper/2401.13218/) (2024)
- [LLMs are not Zero-Shot Reasoners for Biomedical Information Extraction](https://mlsanity.com/paper/2408.12249/) (2024)
- [NAT-NL2GQL: A Novel Multi-Agent Framework for Translating Natural Language to Graph Query Language](https://mlsanity.com/paper/2412.10434/) (2024)
- [FsPONER: Few-shot Prompt Optimization for Named Entity Recognition in Domain-specific Scenarios](https://mlsanity.com/paper/2407.08035/) (2024)
- [PaDeLLM-NER: Parallel Decoding in Large Language Models for Named Entity Recognition](https://mlsanity.com/paper/2402.04838/) (2024)
- [Leveraging Large Language Models for Enhanced NLP Task Performance through Knowledge Distillation and Optimized Training Strategies](https://mlsanity.com/paper/2402.09282/) (2024)
- [A Cooperative Multi-Agent Framework for Zero-Shot Named Entity Recognition](https://mlsanity.com/paper/2502.18702/) (2025)
- [P-ICL: Point In-Context Learning for Named Entity Recognition with Large Language Models](https://mlsanity.com/paper/2405.04960/) (2024)
- [Beyond Boundaries: Learning a Universal Entity Taxonomy across Datasets and Languages for Open Named Entity Recognition](https://mlsanity.com/paper/2406.11192/) (2024)
- [DynamicNER: A Dynamic, Multilingual, and Fine-Grained Dataset for LLM-based Named Entity Recognition](https://mlsanity.com/paper/2409.11022/) (2024)
- [Evaluating Zero-Shot Multilingual Aspect-Based Sentiment Analysis with Large Language Models](https://mlsanity.com/paper/2412.12564/) (2024)
- [llmNER: (Zero|Few)-Shot Named Entity Recognition, Exploiting the Power of Large Language Models](https://mlsanity.com/paper/2406.04528/) (2024)
- [Quantifying Geospatial in the Common Crawl Corpus](https://mlsanity.com/paper/2406.04952/) (2024)
- [Retrieval Augmented Instruction Tuning for Open NER with Large Language Models](https://mlsanity.com/paper/2406.17305/) (2024)
- [Entity Decomposition with Filtering: A Zero-Shot Clinical Named Entity Recognition Framework](https://mlsanity.com/paper/2407.04629/) (2024)
- [Towards Efficient Resume Understanding: A Multi-Granularity Multi-Modal Pre-Training Approach](https://mlsanity.com/paper/2404.13067/) (2024)
- [BANER: Boundary-Aware LLMs for Few-Shot Named Entity Recognition](https://mlsanity.com/paper/2412.02228/) (2024)
- [Selecting and Merging: Towards Adaptable and Scalable Named Entity Recognition with Large Language Models](https://mlsanity.com/paper/2506.22813/) (2025)
- [Improving Table Understanding with LLMs and Entity-Oriented Search](https://mlsanity.com/paper/2508.17028/) (2025)
- [OpenNER 1.0: Standardized Open-Access Named Entity Recognition Datasets in 50+ Languages](https://mlsanity.com/paper/2412.09587/) (2024)
- [Beyond path selection: Better LLMs for Scientific Information Extraction with MimicSFT and Relevance and Rule-induced(R2)GRPO](https://mlsanity.com/paper/2505.22068/) (2025)
- [EL4NER: Ensemble Learning for Named Entity Recognition via Multiple Small-Parameter Large Language Models](https://mlsanity.com/paper/2505.23038/) (2025)

## Abstract (arXiv)

Large language models (LLMs) exhibited powerful capability in various natural language processing tasks. This work focuses on exploring LLM performance on zero-shot information extraction, with a focus on the ChatGPT and named entity recognition (NER) task. Inspired by the remarkable reasoning capability of LLM on symbolic and arithmetic reasoning, we adapt the prevalent reasoning methods to NER and propose reasoning strategies tailored for NER. First, we explore a decomposed question-answering paradigm by breaking down the NER task into simpler subproblems by labels. Second, we propose syntactic augmentation to stimulate the model's intermediate thinking in two ways: syntactic prompting, which encourages the model to analyze the syntactic structure itself, and tool augmentation, which provides the model with the syntactic information generated by a parsing tool. Besides, we adapt self-consistency to NER by proposing a two-stage majority voting strategy, which first votes for the most consistent mentions, then the most consistent types. The proposed methods achieve remarkable improvements for zero-shot NER across seven benchmarks, including Chinese and English datasets, and on both domain-specific and general-domain scenarios. In addition, we present a comprehensive analysis of the error types with suggestions for optimization directions. We also verify the effectiveness of the proposed methods on the few-shot setting and other LLMs.