# Reconciliation of Pre-trained Models and Prototypical Neural Networks in Few-shot Named Entity Recognition

arXiv: https://arxiv.org/abs/2211.03270 · HTML: https://mlsanity.com/paper/2211.03270/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 29 of its 50 references that are in the map.
- First posted on arXiv 7 Nov 2022.

## Builds on (most cited first, 29 of 29)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [Representation Learning with Contrastive Predictive Coding](https://mlsanity.com/paper/1807.03748/) (2018)
- [Efficient Estimation of Word Representations in Vector Space](https://mlsanity.com/paper/1301.3781/) (2013)
- [Prototypical Networks for Few-shot Learning](https://mlsanity.com/paper/1703.05175/) (2017)
- [Matching Networks for One Shot Learning](https://mlsanity.com/paper/1606.04080/) (2016)
- [Learning to Compare: Relation Network for Few-Shot Learning](https://mlsanity.com/paper/1711.06025/) (2017)
- [Aligning Books and Movies: Towards Story-like Visual Explanations by Watching Movies and Reading Books](https://mlsanity.com/paper/1506.06724/) (2015)
- [MultiWOZ -- A Large-Scale Multi-Domain Wizard-of-Oz Dataset for Task-Oriented Dialogue Modelling](https://mlsanity.com/paper/1810.00278/) (2018)
- [Neural Architectures for Named Entity Recognition](https://mlsanity.com/paper/1603.01360/) (2016)
- [TADAM: Task dependent adaptive metric for improved few-shot learning](https://mlsanity.com/paper/1805.10123/) (2018)
- [Dynamic Few-Shot Visual Learning without Forgetting](https://mlsanity.com/paper/1804.09458/) (2018)
- [Snips Voice Platform: an embedded Spoken Language Understanding system for private-by-design voice interfaces](https://mlsanity.com/paper/1805.10190/) (2018)
- [A Survey of Unsupervised Deep Domain Adaptation](https://mlsanity.com/paper/1812.02849/) (2018)
- [Representation Degeneration Problem in Training Natural Language Generation Models](https://mlsanity.com/paper/1907.12009/) (2019)
- [All-but-the-Top: Simple and Effective Postprocessing for Word Representations](https://mlsanity.com/paper/1702.01417/) (2017)
- [Free Lunch for Few-shot Learning: Distribution Calibration](https://mlsanity.com/paper/2101.06395/) (2021)
- [Breaking the Softmax Bottleneck: A High-Rank RNN Language Model](https://mlsanity.com/paper/1711.03953/) (2017)
- [Few-NERD: A Few-Shot Named Entity Recognition Dataset](https://mlsanity.com/paper/2105.07464/) (2021)
- [A Survey on Recent Advances in Named Entity Recognition from Deep Learning models](https://mlsanity.com/paper/1910.11470/) (2019)
- [Visual Domain Adaptation with Manifold Embedded Distribution Alignment](https://mlsanity.com/paper/1807.07258/) (2018)
- [Boosting Few-Shot Learning With Adaptive Margin Loss](https://mlsanity.com/paper/2005.13826/) (2020)
- [CONTaiNER: Few-Shot Named Entity Recognition via Contrastive Learning](https://mlsanity.com/paper/2109.07589/) (2021)
- [Few-shot Text Classification with Distributional Signatures](https://mlsanity.com/paper/1908.06039/) (2019)
- [Exploring Task Difficulty for Few-Shot Relation Extraction](https://mlsanity.com/paper/2109.05473/) (2021)
- [Learning from Miscellaneous Other-Class Words for Few-shot Named Entity Recognition](https://mlsanity.com/paper/2106.15167/) (2021)
- [Squeezing Backbone Feature Distributions to the Max for Efficient Few-Shot Learning](https://mlsanity.com/paper/2110.09446/) (2021)
- [Learning to Learn to Disambiguate: Meta-Learning for Few-Shot Word Sense Disambiguation](https://mlsanity.com/paper/2004.14355/) (2020)

## Built on by (most cited first, 1 of 1)

- [DE3-BERT: Distance-Enhanced Early Exiting for BERT based on Prototypical Networks](https://mlsanity.com/paper/2402.05948/) (2024)

## Abstract (arXiv)

Incorporating large-scale pre-trained models with the prototypical neural networks is a de-facto paradigm in few-shot named entity recognition. Existing methods, unfortunately, are not aware of the fact that embeddings from pre-trained models contain a prominently large amount of information regarding word frequencies, biasing prototypical neural networks against learning word entities. This discrepancy constrains the two models' synergy. Thus, we propose a one-line-code normalization method to reconcile such a mismatch with empirical and theoretical grounds. Our experiments based on nine benchmark datasets show the superiority of our method over the counterpart models and are comparable to the state-of-the-art methods. In addition to the model enhancement, our work also provides an analytical viewpoint for addressing the general problems in few-shot name entity recognition or other tasks that rely on pre-trained models or prototypical neural networks.