# Coarse-to-Fine Pre-training for Named Entity Recognition

arXiv: https://arxiv.org/abs/2010.08210 · HTML: https://mlsanity.com/paper/2010.08210/

## Key facts

- Cited by 28 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 16 Oct 2020.

## Built on by (most cited first, 28 of 28)

- [Unified Structure Generation for Universal Information Extraction](https://mlsanity.com/paper/2203.12277/) (2022)
- [Large Language Models are Superpositions of All Characters: Attaining Arbitrary Role-play via Self-Alignment](https://mlsanity.com/paper/2401.12474/) (2024)
- [NuNER: Entity Recognition Encoder Pre-training via LLM-Annotated Data](https://mlsanity.com/paper/2402.15343/) (2024)
- [Accelerating BERT Inference for Sequence Labeling via Early-Exit](https://mlsanity.com/paper/2105.13878/) (2021)
- [Optimizing Bi-Encoder for Named Entity Recognition via Contrastive Learning](https://mlsanity.com/paper/2208.14565/) (2022)
- [Parallel Instance Query Network for Named Entity Recognition](https://mlsanity.com/paper/2203.10545/) (2022)
- [YAYI-UIE: A Chat-Enhanced Instruction Tuning Framework for Universal Information Extraction](https://mlsanity.com/paper/2312.15548/) (2023)
- [MINER: Improving Out-of-Vocabulary Named Entity Recognition from an Information Theoretic Perspective](https://mlsanity.com/paper/2204.04391/) (2022)
- [PromptNER: Prompt Locating and Typing for Named Entity Recognition](https://mlsanity.com/paper/2305.17104/) (2023)
- [Few-shot Named Entity Recognition with Self-describing Networks](https://mlsanity.com/paper/2203.12252/) (2022)
- [Rethinking Negative Instances for Generative Named Entity Recognition](https://mlsanity.com/paper/2402.16602/) (2024)
- [Domain-Specific NER via Retrieving Correlated Samples](https://mlsanity.com/paper/2208.12995/) (2022)
- [Probing Pre-trained Auto-regressive Language Models for Named Entity Typing and Recognition](https://mlsanity.com/paper/2108.11857/) (2021)
- [Information Extraction in Low-Resource Scenarios: Survey and Perspective](https://mlsanity.com/paper/2202.08063/) (2022)
- [Span Classification with Structured Information for Disfluency Detection in Spoken Utterances](https://mlsanity.com/paper/2203.16028/) (2022)
- [NER-MQMRC: Formulating Named Entity Recognition as Multi Question Machine Reading Comprehension](https://mlsanity.com/paper/2205.05904/) (2022)
- [SEE-Few: Seed, Expand and Entail for Few-shot Named Entity Recognition](https://mlsanity.com/paper/2210.05632/) (2022)
- [Causal Document-Grounded Dialogue Pre-training](https://mlsanity.com/paper/2305.10927/) (2023)
- [Emotion-Cause Pair Extraction as Question Answering](https://mlsanity.com/paper/2301.01982/) (2023)
- [A Confidence-based Partial Label Learning Model for Crowd-Annotated Named Entity Recognition](https://mlsanity.com/paper/2305.12485/) (2023)
- [VerifiNER: Verification-augmented NER via Knowledge-grounded Reasoning with Large Language Models](https://mlsanity.com/paper/2402.18374/) (2024)
- [Mitigating Out-of-Entity Errors in Named Entity Recognition: A Sentence-Level Strategy](https://mlsanity.com/paper/2412.08434/) (2024)
- [Self-Adaptive Named Entity Recognition by Retrieving Unstructured Knowledge](https://mlsanity.com/paper/2210.07523/) (2022)
- [IXA/Cogcomp at SemEval-2023 Task 2: Context-enriched Multilingual Named Entity Recognition using Knowledge Bases](https://mlsanity.com/paper/2304.10637/) (2023)
- [Towards Safer Operations: An Expert-involved Dataset of High-Pressure Gas Incidents for Preventing Future Failures](https://mlsanity.com/paper/2310.12074/) (2023)
- [Information Extraction from Documents: Question Answering vs Token Classification in real-world setups](https://mlsanity.com/paper/2304.10994/) (2023)
- [NER-to-MRC: Named-Entity Recognition Completely Solving as Machine Reading Comprehension](https://mlsanity.com/paper/2305.03970/) (2023)
- [Enhancing AI Research Paper Analysis: Methodology Component Extraction using Factored Transformer-based Sequence Modeling Approach](https://mlsanity.com/paper/2311.03401/) (2023)

## Abstract (arXiv)

More recently, Named Entity Recognition hasachieved great advances aided by pre-trainingapproaches such as BERT. However, currentpre-training techniques focus on building lan-guage modeling objectives to learn a gen-eral representation, ignoring the named entity-related knowledge. To this end, we proposea NER-specific pre-training framework to in-ject coarse-to-fine automatically mined entityknowledge into pre-trained models. Specifi-cally, we first warm-up the model via an en-tity span identification task by training it withWikipedia anchors, which can be deemed asgeneral-typed entities. Then we leverage thegazetteer-based distant supervision strategy totrain the model extract coarse-grained typedentities. Finally, we devise a self-supervisedauxiliary task to mine the fine-grained namedentity knowledge via clustering.Empiricalstudies on three public NER datasets demon-strate that our framework achieves significantimprovements against several pre-trained base-lines, establishing the new state-of-the-art per-formance on three benchmarks. Besides, weshow that our framework gains promising re-sults without using human-labeled trainingdata, demonstrating its effectiveness in label-few and low-resource scenarios