mlsanityData to 7 Oct 2026

Semantic-Oriented Unlabeled Priming for Large-Scale Language Models

arXiv 2202.06133 ↗First posted 12 Feb 2022cs.CLcited by 12

Not on the map. Why?

Abstract (arXiv)

Due to the high costs associated with finetuning large language models, various recent works propose to adapt them to specific tasks without any parameter updates through in-context learning. Unfortunately, for in-context learning there is currently no way to leverage unlabeled data, which is often much easier to obtain in large quantities than labeled examples. In this work, we therefore investigate ways to make use of unlabeled examples to improve the zero-shot performance of pretrained language models without any finetuning: We introduce Semantic-Oriented Unlabeled Priming (SOUP), a method that classifies examples by retrieving semantically similar unlabeled examples, assigning labels to them in a zero-shot fashion, and then using them for in-context learning. We also propose bag-of-contexts priming, a new priming strategy that is more suitable for our setting and enables the usage of more examples than fit into the context window.

Key facts. Cited by 12 papers in the map (data through 7 Oct 2026). Builds on 23 of its 34 references that are in the map. First posted on arXiv 12 Feb 2022.

Builds on

23 of 34 references.

References · 23

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  4. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  5. 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,049
  6. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,014
  7. 2019ALBERT: A Lite BERT for Self-supervised Learning of Language Representationscited by 2,772
  8. 2015Character-level Convolutional Networks for Text Classificationcited by 2,293
  9. 2021SimCSE: Simple Contrastive Learning of Sentence Embeddingscited by 2,168
  10. 2016MS MARCO: A Human Generated MAchine Reading COmprehension Datasetcited by 1,591
  11. 2020Don't Stop Pretraining: Adapt Language Models to Domains and Taskscited by 1,548
  12. 2021Multitask Prompted Training Enables Zero-Shot Task Generalizationcited by 1,462
11 more
  1. 2020Making Pre-trained Language Models Better Few-shot Learnerscited by 1,255
  2. 2019Energy and Policy Considerations for Deep Learning in NLPcited by 1,242
  3. 2021What Makes Good In-Context Examples for GPT-3?cited by 1,184
  4. 2021Fantastically Ordered Prompts and Where to Find Them: Overcoming Few-Shot Prompt Order Sensitivitycited by 1,122
  5. 2019Unsupervised Data Augmentation for Consistency Trainingcited by 1,057
  6. 2020Exploiting Cloze Questions for Few Shot Text Classification and Natural Language Inferencecited by 934
  7. 2020It's Not Just Size That Matters: Small Language Models Are Also Few-Shot Learnerscited by 668
  8. 2020How Can We Know When Language Models Know? On the Calibration of Language Models for Question Answeringcited by 439
  9. 2021Self-Diagnosis and Self-Debiasing: A Proposal for Reducing Corpus-Based Bias in NLPcited by 336
  10. 2021Noisy Channel Language Model Prompting for Few-Shot Text Classificationcited by 163
  11. 2021Reordering Examples Helps during Priming-based Few-Shot Learningcited by 50

Built on by

Cited by year

Citing papers per year: 2022: 2; 2023: 8; 2024: 2.

12 papers.

  1. 2022Z-ICL: Zero-Shot In-Context Learning with Pseudo-Demonstrationscited by 70
  2. 2022Cross-Lingual Retrieval Augmented Prompt for Low-Resource Languagescited by 17
  3. 2023From Classification to Generation: Insights into Crosslingual Retrieval Augmented ICLcited by 17
  4. 2023Confronting LLMs with Traditional ML: Rethinking the Fairness of Large Language Models in Tabular Classificationscited by 15
  5. 2024Assessing and Verifying Task Utility in LLM-Powered Applicationscited by 15
  6. 2023Retrieval-augmented Multilingual Knowledge Editingcited by 13
  7. 2023DADA: Dialect Adaptation via Dynamic Aggregation of Linguistic Rulescited by 12
  8. 2023Is Prompt-Based Finetuning Always Better than Vanilla Finetuning? Insights from Cross-Lingual Language Understandingcited by 7
  9. 2023Crosslingual Retrieval Augmented In-context Learning for Banglacited by 7
  10. 2024Towards better Human-Agent Alignment: Assessing Task Utility in LLM-Powered Applicationscited by 5
  11. 2023Decoding Susceptibility: Modeling Misbelief to Misinformation Through a Computational Approachcited by 3
  12. 2023SMoA: Sparse Mixture of Adapters to Mitigate Multiple Dataset Biasescited by 2