mlsanityData to 7 Oct 2026

Contrastive Demonstration Tuning for Pre-trained Language Models

arXiv 2204.04392 ↗First posted 9 Apr 2022cs.CL cs.AI cs.IRcited by 7

Not on the map. Why?

Abstract (arXiv)

Pretrained language models can be effectively stimulated by textual prompts or demonstrations, especially in low-data scenarios. Recent works have focused on automatically searching discrete or continuous prompts or optimized verbalizers, yet studies for the demonstration are still limited. Concretely, the demonstration examples are crucial for an excellent final performance of prompt-tuning. In this paper, we propose a novel pluggable, extensible, and efficient approach named contrastive demonstration tuning, which is free of demonstration sampling. Furthermore, the proposed approach can be: (i) Plugged into any previous prompt-tuning approaches; (ii) Extended to widespread classification tasks with a large number of categories. Experimental results on 16 datasets illustrate that our method integrated with previous approaches LM-BFF and P-tuning can yield better performance. Code is available in https://github.com/zjunlp/PromptKG/tree/main/research/Demo-Tuning.

Key facts. Cited by 7 papers in the map (data through 7 Oct 2026). Builds on 55 of its 75 references that are in the map. First posted on arXiv 9 Apr 2022.

Builds on

55 of 75 references.

References · 55

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
  4. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  5. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 11,979
  6. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,160
  7. 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,049
  8. 2019BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehensioncited by 5,874
  9. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,854
  10. 2020Bootstrap your own latent: A new approach to self-supervised Learningcited by 4,712
  11. 2021Prefix-Tuning: Optimizing Continuous Prompts for Generationcited by 3,878
  12. 2021The Power of Scale for Parameter-Efficient Prompt Tuningcited by 3,769
38 more
  1. 2013Distributed Representations of Words and Phrases and their Compositionalitycited by 3,462
  2. 2017Prototypical Networks for Few-shot Learningcited by 3,001
  3. 2020Exploring Simple Siamese Representation Learningcited by 2,853
  4. 2021Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processingcited by 2,222
  5. 2021SimCSE: Simple Contrastive Learning of Sentence Embeddingscited by 2,168
  6. 2020REALM: Retrieval-Augmented Language Model Pre-Trainingcited by 2,126
  7. 2022Rethinking the Role of Demonstrations: What Makes In-Context Learning Work?cited by 1,413
  8. 2020Making Pre-trained Language Models Better Few-shot Learnerscited by 1,255
  9. 2021What Makes Good In-Context Examples for GPT-3?cited by 1,184
  10. 2020Exploiting Cloze Questions for Few Shot Text Classification and Natural Language Inferencecited by 934
  11. 2021GPT Understands, Toocited by 715
  12. 2020Self-supervised Learning: Generative or Contrastivecited by 693
  13. 2021P-Tuning v2: Prompt Tuning Can Be Comparable to Fine-tuning Universally Across Scales and Taskscited by 686
  14. 2019Unified Language Model Pre-training for Natural Language Understanding and Generationcited by 668
  15. 2020It's Not Just Size That Matters: Small Language Models Are Also Few-Shot Learnerscited by 668
  16. 2020A Survey on Contrastive Self-supervised Learningcited by 553
  17. 2021Learning How to Ask: Querying LMs with Mixtures of Soft Promptscited by 376
  18. 2021Factual Probing Is [MASK]: Learning vs. Learning to Recallcited by 291
  19. 2020DeCLUTR: Deep Contrastive Learning for Unsupervised Textual Representationscited by 249
  20. 2021PTR: Prompt Tuning with Rules for Text Classificationcited by 238
  21. 2021SPoT: Better Frozen Model Adaptation through Soft Prompt Transfercited by 235
  22. 2021ConSERT: A Contrastive Framework for Self-Supervised Sentence Representation Transfercited by 231
  23. 2021WARP: Word-level Adversarial ReProgrammingcited by 220
  24. 2020UniLMv2: Pseudo-Masked Language Models for Unified Language Model Pre-Trainingcited by 204
  25. 2018An efficient framework for learning sentence representationscited by 197
  26. 2021KnowPrompt: Knowledge-aware Prompt-tuning with Synergistic Optimization for Relation Extractioncited by 180
  27. 2021Template-Based Named Entity Recognition Using BARTcited by 153
  28. 2021Entailment as Few-Shot Learnercited by 122
  29. 2021Differentiable Prompt Makes Pre-trained Language Models Better Few-shot Learnerscited by 115
  30. 2021Self-Guided Contrastive Learning for BERT Sentence Representationscited by 103
  31. 2021Improving and Simplifying Pattern Exploiting Trainingcited by 97
  32. 2021Template-free Prompt Tuning for Few-shot NERcited by 73
  33. 2022OntoProtein: Protein Pretraining With Gene Ontology Embeddingcited by 70
  34. 2021Prompt-Learning for Fine-Grained Entity Typingcited by 62
  35. 2022From Discrimination to Generation: Knowledge Graph Completion with Generative Transformercited by 45
  36. 2021Good Examples Make A Faster Learner: Simple Demonstration-based Learning for Low-resource NERcited by 38
  37. 2021MSP: Multi-Stage Prompting for Making Pre-trained Language Models Better Translatorscited by 35
  38. 2022Ontology-enhanced Prompt-tuning for Few-shot Learningcited by 24

50 most cited of 55 All 55 (JSON)

Built on by

Cited by year

Citing papers per year: 2022: 3; 2023: 2; 2024: 2.

7 papers.

  1. 2024InstructEdit: Instruction-based Knowledge Editing for Large Language Modelscited by 17
  2. 2022Robustness of Demonstration-based Learning Under Limited Data Scenariocited by 11
  3. 2023A Multi-Task Semantic Decomposition Framework with Task-specific Pre-training for Few-Shot NERcited by 9
  4. 2022Good Visual Guidance Makes A Better Extractor: Hierarchical Visual Prefix for Multimodal Entity and Relation Extractioncited by 7
  5. 2023DemoNSF: A Multi-task Demonstration-based Generative Framework for Noisy Slot Filling Taskcited by 5
  6. 2024Tuning LLMs with Contrastive Alignment Instructions for Machine Translation in Unseen, Low-resource Languagescited by 3
  7. 2022Let Me Check the Examples: Enhancing Demonstration Learning via Explicit Imitationcited by 1