# Probing Pre-trained Auto-regressive Language Models for Named Entity Typing and Recognition

arXiv: https://arxiv.org/abs/2108.11857 · HTML: https://mlsanity.com/paper/2108.11857/

## Key facts

- Cited by 10 papers in the map (data through 14 Oct 2026).
- Builds on 21 of its 35 references that are in the map.
- First posted on arXiv 26 Aug 2021.

## Builds on (most cited first, 21 of 21)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [Language Models as Knowledge Bases?](https://mlsanity.com/paper/1909.01066/) (2019)
- [Calibrate Before Use: Improving Few-Shot Performance of Language Models](https://mlsanity.com/paper/2102.09690/) (2021)
- [How Can We Know What Language Models Know?](https://mlsanity.com/paper/1911.12543/) (2019)
- [The Secret Sharer: Evaluating and Testing Unintended Memorization in Neural Networks](https://mlsanity.com/paper/1802.08232/) (2018)
- [A Unified MRC Framework for Named Entity Recognition](https://mlsanity.com/paper/1910.11476/) (2019)
- [Few-NERD: A Few-Shot Named Entity Recognition Dataset](https://mlsanity.com/paper/2105.07464/) (2021)
- [A Survey on Recent Advances in Named Entity Recognition from Deep Learning models](https://mlsanity.com/paper/1910.11470/) (2019)
- [Learning Named Entity Tagger using Domain-Specific Dictionary](https://mlsanity.com/paper/1809.03599/) (2018)
- [Named Entity Recognition without Labelled Data: A Weak Supervision Approach](https://mlsanity.com/paper/2004.14723/) (2020)
- ["You are grounded!": Latent Name Artifacts in Pre-trained Language Models](https://mlsanity.com/paper/2004.03012/) (2020)
- [Rethinking Generalization of Neural Models: A Named Entity Recognition Case Study](https://mlsanity.com/paper/2001.03844/) (2020)
- [TriggerNER: Learning with Entity Triggers as Explanations for Named Entity Recognition](https://mlsanity.com/paper/2004.07493/) (2020)
- [Coarse-to-Fine Pre-training for Named Entity Recognition](https://mlsanity.com/paper/2010.08210/) (2020)
- [Zero-Shot Open Entity Typing as Type-Compatible Grounding](https://mlsanity.com/paper/1907.03228/) (2019)
- [A Multi-task Approach for Named Entity Recognition in Social Media Data](https://mlsanity.com/paper/1906.04135/) (2019)
- [Contextualized Embeddings in Named-Entity Recognition: An Empirical Study on Generalization](https://mlsanity.com/paper/2001.08053/) (2020)
- [MZET: Memory Augmented Zero-Shot Fine-grained Named Entity Typing](https://mlsanity.com/paper/2004.01267/) (2020)
- [Modeling Noisiness to Recognize Named Entities using Multitask Neural Networks on Social Media](https://mlsanity.com/paper/1906.04129/) (2019)

## Built on by (most cited first, 10 of 10)

- [NBIAS: A Natural Language Processing Framework for Bias Identification in Text](https://mlsanity.com/paper/2308.01681/) (2023)
- [Understanding and Improving Information Preservation in Prompt Compression for LLMs](https://mlsanity.com/paper/2503.19114/) (2025)
- [A Human Subject Study of Named Entity Recognition (NER) in Conversational Music Recommendation Queries](https://mlsanity.com/paper/2303.06944/) (2023)
- [Large language models and linguistic intentionality](https://mlsanity.com/paper/2404.09576/) (2024)
- [A Benchmark and Robustness Study of In-Context-Learning with Large Language Models in Music Entity Detection](https://mlsanity.com/paper/2412.11851/) (2024)
- [Embedded Named Entity Recognition using Probing Classifiers](https://mlsanity.com/paper/2403.11747/) (2024)
- [Large-Scale Label Interpretation Learning for Few-Shot Named Entity Recognition](https://mlsanity.com/paper/2403.14222/) (2024)
- [Do Language Models Learn about Legal Entity Types during Pretraining?](https://mlsanity.com/paper/2310.13092/) (2023)
- [GPT-3 Powered Information Extraction for Building Robust Knowledge Bases](https://mlsanity.com/paper/2408.04641/) (2024)
- [Beyond Musical Descriptors: Extracting Preference-Bearing Intent in Music Queries](https://mlsanity.com/paper/2602.12301/) (2026)

## Abstract (arXiv)

Despite impressive results of language models for named entity recognition (NER), their generalization to varied textual genres, a growing entity type set, and new entities remains a challenge. Collecting thousands of annotations in each new case for training or fine-tuning is expensive and time-consuming. In contrast, humans can easily identify named entities given some simple instructions. Inspired by this, we challenge the reliance on large datasets and study pre-trained language models for NER in a meta-learning setup. First, we test named entity typing (NET) in a zero-shot transfer scenario. Then, we perform NER by giving few examples at inference. We propose a method to select seen and rare / unseen names when having access only to the pre-trained model and report results on these groups. The results show: auto-regressive language models as meta-learners can perform NET and NER fairly well especially for regular or seen names; name irregularity when often present for a certain entity type can become an effective exploitable cue; names with words foreign to the model have the most negative impact on results; the model seems to rely more on name than context cues in few-shot NER.