mlsanityData to 7 Oct 2026

Pre-trained Token-replaced Detection Model as Few-shot Learner

arXiv 2203.03235 ↗First posted 7 Mar 2022cs.CL cs.AIcited by 7

Not on the map. Why?

Abstract (arXiv)

Pre-trained masked language models have demonstrated remarkable ability as few-shot learners. In this paper, as an alternative, we propose a novel approach to few-shot learning with pre-trained token-replaced detection models like ELECTRA. In this approach, we reformulate a classification or a regression task as a token-replaced detection problem. Specifically, we first define a template and label description words for each task and put them into the input to form a natural language prompt. Then, we employ the pre-trained token-replaced detection model to predict which label description word is the most original (i.e., least replaced) among all label description words in the prompt. A systematic evaluation on 16 datasets demonstrates that our approach outperforms few-shot learners with pre-trained masked language models in both one-sentence and two-sentence learning tasks.

Key facts. Cited by 7 papers in the map (data through 7 Oct 2026). Builds on 31 of its 41 references that are in the map. First posted on arXiv 7 Mar 2022.

Builds on

31 of 41 references.

References · 31

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  4. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  5. 2022Generative Adversarial Networkscited by 5,603
  6. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,854
  7. 2015A large annotated corpus for learning natural language inferencecited by 1,837
  8. 2021Calibrate Before Use: Improving Few-Shot Performance of Language Modelscited by 1,371
  9. 2020Making Pre-trained Language Models Better Few-shot Learnerscited by 1,255
  10. 2021What Makes Good In-Context Examples for GPT-3?cited by 1,184
  11. 2021Fantastically Ordered Prompts and Where to Find Them: Overcoming Few-Shot Prompt Order Sensitivitycited by 1,122
  12. 2019How Can We Know What Language Models Know?cited by 953
19 more
  1. 2018Neural Network Acceptability Judgmentscited by 952
  2. 2020Exploiting Cloze Questions for Few Shot Text Classification and Natural Language Inferencecited by 934
  3. 2021GPT Understands, Toocited by 715
  4. 2020It's Not Just Size That Matters: Small Language Models Are Also Few-Shot Learnerscited by 668
  5. 2020Fine-Tuning Pretrained Language Models: Weight Initializations, Data Orders, and Early Stoppingcited by 377
  6. 2021Learning How to Ask: Querying LMs with Mixtures of Soft Promptscited by 376
  7. 2021PTR: Prompt Tuning with Rules for Text Classificationcited by 238
  8. 2020Revisiting Few-sample BERT Fine-tuningcited by 237
  9. 2021WARP: Word-level Adversarial ReProgrammingcited by 220
  10. 2021Noisy Channel Language Model Prompting for Few-Shot Text Classificationcited by 163
  11. 2021Entailment as Few-Shot Learnercited by 122
  12. 2021Differentiable Prompt Makes Pre-trained Language Models Better Few-shot Learnerscited by 115
  13. 2021BERTese: Learning to Speak to BERTcited by 82
  14. 2019Learning and Evaluating General Linguistic Intelligencecited by 78
  15. 2022Corrupted Image Modeling for Self-Supervised Visual Pre-Trainingcited by 68
  16. 2019Zero-shot Text Classification With Generative Language Modelscited by 55
  17. 2021Efficient Passage Retrieval with Hashing for Open-domain Question Answeringcited by 53
  18. 2021NSP-BERT: A Prompt-based Few-Shot Learner Through an Original Pre-training Task--Next Sentence Predictioncited by 23
  19. 2021ASR Rescoring and Confidence Estimation with ELECTRAcited by 16

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 3; 2023: 2; 2024: 1; 2025: 0; 2026: 1.

7 papers.

  1. 2023PIEClass: Weakly-Supervised Text Classification with Prompting and Noise-Robust Iterative Ensemble Trainingcited by 6
  2. 2022ELECTRA is a Zero-Shot Learner, Toocited by 4
  3. 2023Reformulating NLP tasks to Capture Longitudinal Manifestation of Language Disorders in People with Dementiacited by 4
  4. 2024Transferable and Forecastable User Targeting Foundation Modelcited by 4
  5. 2022Prompting ELECTRA: Few-Shot Learning with Discriminative Pre-Trained Modelscited by 2
  6. 2022Let Me Check the Examples: Enhancing Demonstration Learning via Explicit Imitationcited by 1
  7. 2026LaDA-Band: Language Diffusion Models for Vocal-to-Accompaniment Generationcited by 0