# MCML: A Novel Memory-based Contrastive Meta-Learning Method for Few Shot Slot Tagging

arXiv: https://arxiv.org/abs/2108.11635 · HTML: https://mlsanity.com/paper/2108.11635/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 35 of its 47 references that are in the map.
- First posted on arXiv 26 Aug 2021.

## Builds on (most cited first, 30 of 35)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [A Simple Framework for Contrastive Learning of Visual Representations](https://mlsanity.com/paper/2002.05709/) (2020)
- [Momentum Contrast for Unsupervised Visual Representation Learning](https://mlsanity.com/paper/1911.05722/) (2019)
- [Prototypical Networks for Few-shot Learning](https://mlsanity.com/paper/1703.05175/) (2017)
- [Matching Networks for One Shot Learning](https://mlsanity.com/paper/1606.04080/) (2016)
- [Meta-Learning in Neural Networks: A Survey](https://mlsanity.com/paper/2004.05439/) (2020)
- [An Empirical Study of Example Forgetting during Deep Neural Network Learning](https://mlsanity.com/paper/1812.05159/) (2018)
- [Snips Voice Platform: an embedded Spoken Language Understanding system for private-by-design voice interfaces](https://mlsanity.com/paper/1805.10190/) (2018)
- [Rapid Learning or Feature Reuse? Towards Understanding the Effectiveness of MAML](https://mlsanity.com/paper/1909.09157/) (2019)
- [Meta-Transfer Learning for Few-Shot Learning](https://mlsanity.com/paper/1812.02391/) (2018)
- [BERT for Joint Intent Classification and Slot Filling](https://mlsanity.com/paper/1902.10909/) (2019)
- [CrossTransformers: spatially-aware few-shot transfer](https://mlsanity.com/paper/2007.11498/) (2020)
- [Supporting Clustering with Contrastive Learning](https://mlsanity.com/paper/2103.12953/) (2021)
- [Few-shot Slot Tagging with Collapsed Dependency Transfer and Label-enhanced Task-adaptive Projection Network](https://mlsanity.com/paper/2006.05702/) (2020)
- [Aspect Level Sentiment Classification with Deep Memory Network](https://mlsanity.com/paper/1605.08900/) (2016)
- [Memory Matching Networks for One-Shot Image Recognition](https://mlsanity.com/paper/1804.08281/) (2018)
- [Few-shot classification in Named Entity Recognition Task](https://mlsanity.com/paper/1812.06158/) (2018)
- [Learning to Remember Rare Events](https://mlsanity.com/paper/1703.03129/) (2017)
- [Meta-learning for Few-shot Natural Language Processing: A Survey](https://mlsanity.com/paper/2007.09604/) (2020)
- [Towards Zero-Shot Frame Semantic Parsing for Domain Scaling](https://mlsanity.com/paper/1707.02363/) (2017)
- [A Theoretical Analysis of the Number of Shots in Few-Shot Learning](https://mlsanity.com/paper/1909.11722/) (2019)
- [Robust Zero-Shot Cross-Domain Slot Filling with Example Values](https://mlsanity.com/paper/1906.06870/) (2019)
- [Coach: A Coarse-to-Fine Approach for Cross-domain Slot Filling](https://mlsanity.com/paper/2004.11727/) (2020)
- [Prototypical Representation Learning for Relation Extraction](https://mlsanity.com/paper/2103.11647/) (2021)
- [Learning to Classify Intents and Slot Labels Given a Handful of Examples](https://mlsanity.com/paper/2004.10793/) (2020)
- [LGM-Net: Learning to Generate Matching Networks for Few-Shot Learning](https://mlsanity.com/paper/1905.06331/) (2019)
- [Dynamic Memory Induction Networks for Few-Shot Text Classification](https://mlsanity.com/paper/2005.05727/) (2020)
- [Zero-Shot Adaptive Transfer for Conversational Language Understanding](https://mlsanity.com/paper/1808.10059/) (2018)

## Built on by (most cited first, 4 of 4)

- [Information Extraction in Low-Resource Scenarios: Survey and Perspective](https://mlsanity.com/paper/2202.08063/) (2022)
- [Improving Meta-learning for Low-resource Text Classification and Generation via Memory Imitation](https://mlsanity.com/paper/2203.11670/) (2022)
- [Inconsistent Few-Shot Relation Classification via Cross-Attentional Prototype Networks with Contrastive Learning](https://mlsanity.com/paper/2110.08254/) (2021)
- [Prior Omission of Dissimilar Source Domain(s) for Cost-Effective Few-Shot Learning](https://mlsanity.com/paper/2109.05234/) (2021)

## Abstract (arXiv)

Meta-learning is widely used for few-shot slot tagging in task of few-shot learning. The performance of existing methods is, however, seriously affected by textitsample forgetting issue, where the model forgets the historically learned meta-training tasks while solely relying on support sets when adapting to new tasks. To overcome this predicament, we propose the Memory-based Contrastive Meta-Learning (aka, MCML) method, including textitlearn-from-the-memory and textitadaption-from-the-memory modules, which bridge the distribution gap between training episodes and between training and testing respectively. Specifically, the former uses an explicit memory bank to keep track of the label representations of previously trained episodes, with a contrastive constraint between the label representations in the current episode with the historical ones stored in the memory. In addition, the adaption-from-memory mechanism is introduced to learn more accurate and robust representations based on the shift between the same labels embedded in the testing episodes and memory. Experimental results show that the MCML outperforms several state-of-the-art methods on both SNIPS and NER datasets and demonstrates strong scalability with consistent improvement when the number of shots gets greater.