# CONTaiNER: Few-Shot Named Entity Recognition via Contrastive Learning

arXiv: https://arxiv.org/abs/2109.07589 · HTML: https://mlsanity.com/paper/2109.07589/

## Key facts

- Cited by 64 papers in the map (data through 14 Oct 2026).
- Builds on 29 of its 52 references that are in the map.
- First posted on arXiv 15 Sep 2021.

## Builds on (most cited first, 29 of 29)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [A Simple Framework for Contrastive Learning of Visual Representations](https://mlsanity.com/paper/2002.05709/) (2020)
- [Prototypical Networks for Few-shot Learning](https://mlsanity.com/paper/1703.05175/) (2017)
- [Supervised Contrastive Learning](https://mlsanity.com/paper/2004.11362/) (2020)
- [Deep contextualized word representations](https://mlsanity.com/paper/1802.05365/) (2018)
- [Matching Networks for One Shot Learning](https://mlsanity.com/paper/1606.04080/) (2016)
- [Unsupervised Feature Learning via Non-Parametric Instance-level Discrimination](https://mlsanity.com/paper/1805.01978/) (2018)
- [Self-Supervised Learning of Pretext-Invariant Representations](https://mlsanity.com/paper/1912.01991/) (2019)
- [Neural Architectures for Named Entity Recognition](https://mlsanity.com/paper/1603.01360/) (2016)
- [Bidirectional LSTM-CRF Models for Sequence Tagging](https://mlsanity.com/paper/1508.01991/) (2015)
- [True Few-Shot Learning with Language Models](https://mlsanity.com/paper/2105.11447/) (2021)
- [End-to-end Sequence Labeling via Bi-directional LSTM-CNNs-CRF](https://mlsanity.com/paper/1603.01354/) (2016)
- [Deep Gaussian Embedding of Graphs: Unsupervised Inductive Learning via Ranking](https://mlsanity.com/paper/1707.03815/) (2017)
- [Discriminative Unsupervised Feature Learning with Exemplar Convolutional Neural Networks](https://mlsanity.com/paper/1406.6909/) (2014)
- [FewRel: A Large-Scale Supervised Few-Shot Relation Classification Dataset with State-of-the-Art Evaluation](https://mlsanity.com/paper/1810.10147/) (2018)
- [Unsupervised Embedding Learning via Invariant and Spreading Instance Feature](https://mlsanity.com/paper/1904.03436/) (2019)
- [Local Aggregation for Unsupervised Learning of Visual Embeddings](https://mlsanity.com/paper/1903.12355/) (2019)
- [Multi-task Self-Supervised Visual Learning](https://mlsanity.com/paper/1708.07860/) (2017)
- [SimpleShot: Revisiting Nearest-Neighbor Classification for Few-Shot Learning](https://mlsanity.com/paper/1911.04623/) (2019)
- [Template-Based Named Entity Recognition Using BART](https://mlsanity.com/paper/2106.01760/) (2021)
- [Few-NERD: A Few-Shot Named Entity Recognition Dataset](https://mlsanity.com/paper/2105.07464/) (2021)
- [Word Representations via Gaussian Embedding](https://mlsanity.com/paper/1412.6623/) (2014)
- [Few-shot Slot Tagging with Collapsed Dependency Transfer and Label-enhanced Task-adaptive Projection Network](https://mlsanity.com/paper/2006.05702/) (2020)
- [Few-shot classification in Named Entity Recognition Task](https://mlsanity.com/paper/1812.06158/) (2018)
- [Induction Networks for Few-Shot Text Classification](https://mlsanity.com/paper/1902.10482/) (2019)
- [Few-shot Text Classification with Distributional Signatures](https://mlsanity.com/paper/1908.06039/) (2019)
- [Few-Shot Named Entity Recognition: A Comprehensive Study](https://mlsanity.com/paper/2012.14978/) (2020)
- [Label-Agnostic Sequence Labeling by Copying Nearest Neighbors](https://mlsanity.com/paper/1906.04225/) (2019)
- [Hierarchical Density Order Embeddings](https://mlsanity.com/paper/1804.09843/) (2018)

## Built on by (most cited first, 30 of 64)

- [A Contrastive Framework for Neural Text Generation](https://mlsanity.com/paper/2202.06417/) (2022)
- [State-of-the-art generalisation research in NLP: A taxonomy and review](https://mlsanity.com/paper/2210.03050/) (2022)
- [Language Models Can See: Plugging Visual Controls in Text Generation](https://mlsanity.com/paper/2205.02655/) (2022)
- [A Survey of GPT-3 Family Large Language Models Including ChatGPT and GPT-4](https://mlsanity.com/paper/2310.12321/) (2023)
- [PromptNER: Prompting For Named Entity Recognition](https://mlsanity.com/paper/2305.15444/) (2023)
- [TaCL: Improving BERT Pre-training with Token-aware Contrastive Learning](https://mlsanity.com/paper/2111.04198/) (2021)
- [Modality-Aware Contrastive Instance Learning with Self-Distillation for Weakly-Supervised Audio-Visual Violence Detection](https://mlsanity.com/paper/2207.05500/) (2022)
- [Mask-guided BERT for Few Shot Text Classification](https://mlsanity.com/paper/2302.10447/) (2023)
- [Optimizing Bi-Encoder for Named Entity Recognition via Contrastive Learning](https://mlsanity.com/paper/2208.14565/) (2022)
- [Decomposed Meta-Learning for Few-Shot Named Entity Recognition](https://mlsanity.com/paper/2204.05751/) (2022)
- [Prompt Tuning Pushes Farther, Contrastive Learning Pulls Closer: A Two-Stage Approach to Mitigate Social Biases](https://mlsanity.com/paper/2307.01595/) (2023)
- [Unified Low-Resource Sequence Labeling by Sample-Aware Dynamic Sparse Finetuning](https://mlsanity.com/paper/2311.03748/) (2023)
- [Improving Word Translation via Two-Stage Contrastive Learning](https://mlsanity.com/paper/2203.08307/) (2022)
- [Few-shot Named Entity Recognition with Entity-level Prototypical Network Enhanced by Dispersedly Distributed Prototypes](https://mlsanity.com/paper/2208.08023/) (2022)
- [Few-shot Named Entity Recognition with Self-describing Networks](https://mlsanity.com/paper/2203.12252/) (2022)
- [Improving the Sample Efficiency of Prompt Tuning with Domain Adaptation](https://mlsanity.com/paper/2210.02952/) (2022)
- [How far is Language Model from 100% Few-shot Named Entity Recognition in Medical Domain](https://mlsanity.com/paper/2307.00186/) (2023)
- [mCL-NER: Cross-Lingual Named Entity Recognition via Multi-view Contrastive Learning](https://mlsanity.com/paper/2308.09073/) (2023)
- [Prompt-Based Metric Learning for Few-Shot NER](https://mlsanity.com/paper/2211.04337/) (2022)
- [KRACL: Contrastive Learning with Graph Context Modeling for Sparse Knowledge Graph Completion](https://mlsanity.com/paper/2208.07622/) (2022)
- [Distilling Causal Effect from Miscellaneous Other-Class for Continual Named Entity Recognition](https://mlsanity.com/paper/2210.03980/) (2022)
- [Few-Shot Document-Level Event Argument Extraction](https://mlsanity.com/paper/2209.02203/) (2022)
- [VicunaNER: Zero/Few-shot Named Entity Recognition using Vicuna](https://mlsanity.com/paper/2305.03253/) (2023)
- [A Multi-Task Semantic Decomposition Framework with Task-specific Pre-training for Few-Shot NER](https://mlsanity.com/paper/2308.14533/) (2023)
- [SpanProto: A Two-stage Span-based Prototypical Network for Few-shot Named Entity Recognition](https://mlsanity.com/paper/2210.09049/) (2022)
- [Improving Self-training for Cross-lingual Named Entity Recognition with Contrastive and Prototype Learning](https://mlsanity.com/paper/2305.13628/) (2023)
- [FactMix: Using a Few Labeled In-domain Examples to Generalize to Cross-domain Named Entity Recognition](https://mlsanity.com/paper/2208.11464/) (2022)
- [Learning In-context Learning for Named Entity Recognition](https://mlsanity.com/paper/2305.11038/) (2023)
- [A Prototypical Semantic Decoupling Method via Joint Contrastive Learning for Few-Shot Name Entity Recognition](https://mlsanity.com/paper/2302.13610/) (2023)
- [Text2MDT: Extracting Medical Decision Trees from Medical Texts](https://mlsanity.com/paper/2401.02034/) (2024)

## Abstract (arXiv)

Named Entity Recognition (NER) in Few-Shot setting is imperative for entity tagging in low resource domains. Existing approaches only learn class-specific semantic features and intermediate representations from source domains. This affects generalizability to unseen target domains, resulting in suboptimal performances. To this end, we present CONTaiNER, a novel contrastive learning technique that optimizes the inter-token distribution distance for Few-Shot NER. Instead of optimizing class-specific attributes, CONTaiNER optimizes a generalized objective of differentiating between token categories based on their Gaussian-distributed embeddings. This effectively alleviates overfitting issues originating from training domains. Our experiments in several traditional test domains (OntoNotes, CoNLL'03, WNUT '17, GUM) and a new large scale Few-Shot NER dataset (Few-NERD) demonstrate that on average, CONTaiNER outperforms previous methods by 3%-13% absolute F1 points while showing consistent performance trends, even in challenging scenarios where previous approaches could not achieve appreciable performance.