mlsanityData to 7 Oct 2026

WARP: Word-level Adversarial ReProgramming

arXiv 2101.00121 ↗First posted 1 Jan 2021cs.CLcited by 220

Not on the map. Why?

Abstract (arXiv)

Transfer learning from pretrained language models recently became the dominant approach for solving many NLP tasks. A common approach to transfer learning for multiple tasks that maximize parameter sharing trains one or more task-specific layers on top of the language model. In this paper, we present an alternative approach based on adversarial reprogramming, which extends earlier work on automatic prompt generation. Adversarial reprogramming attempts to learn task-specific word embeddings that, when concatenated to the input text, instruct the language model to solve the specified task. Using up to 25K trainable parameters per task, this approach outperforms all existing methods with up to 25M trainable parameters on the public leaderboard of the GLUE benchmark. Our method, initialized with task-specific human-readable prompts, also works in a few-shot setting, outperforming GPT-3 on two SuperGLUE tasks with just 32 training samples.

Key facts. Cited by 220 papers in the map (data through 7 Oct 2026). Builds on 23 of its 42 references that are in the map. First posted on arXiv 1 Jan 2021.

Builds on

23 of 42 references.

References · 23

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  4. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,854
  5. 2019Parameter-Efficient Transfer Learning for NLPcited by 4,195
  6. 2016SQuAD: 100,000+ Questions for Machine Comprehension of Textcited by 4,115
  7. 2019ALBERT: A Lite BERT for Self-supervised Learning of Language Representationscited by 2,772
  8. 2017A Broad-Coverage Challenge Corpus for Sentence Understanding through Inferencecited by 2,665
  9. 2018Deep contextualized word representationscited by 2,285
  10. 2020DeBERTa: Decoding-enhanced BERT with Disentangled Attentioncited by 2,157
  11. 2019SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systemscited by 1,718
  12. 2019TinyBERT: Distilling BERT for Natural Language Understandingcited by 1,153
11 more
  1. 2017SemEval-2017 Task 1: Semantic Textual Similarity - Multilingual and Cross-lingual Focused Evaluationcited by 998
  2. 2018Neural Network Acceptability Judgmentscited by 952
  3. 2020Exploiting Cloze Questions for Few Shot Text Classification and Natural Language Inferencecited by 934
  4. 2020AdapterFusion: Non-Destructive Task Composition for Transfer Learningcited by 780
  5. 2020It's Not Just Size That Matters: Small Language Models Are Also Few-Shot Learnerscited by 668
  6. 2019On the Cross-lingual Transferability of Monolingual Representationscited by 575
  7. 2016Google's Multilingual Neural Machine Translation System: Enabling Zero-Shot Translationcited by 467
  8. 2019To Tune or Not to Tune? Adapting Pretrained Representations to Diverse Taskscited by 167
  9. 2018Adversarial Reprogramming of Neural Networkscited by 115
  10. 2016Toward Multilingual Neural Machine Translation with Universal Encoder and Decodercited by 76
  11. 2020Pre-Training Transformers as Energy-Based Cloze Modelscited by 52

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 25; 2022: 63; 2023: 74; 2024: 41; 2025: 10; 2026: 7.

220 papers.

Sort:

By area:

  1. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  2. 2021The Power of Scale for Parameter-Efficient Prompt Tuningcited by 3,769
  3. 2021Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processingcited by 2,222
  4. 2023Automatic Prompt Optimization with "Gradient Descent" and Beam Searchcited by 628
  5. 2021Compacter: Efficient Low-Rank Hypercomplex Adapter Layerscited by 458
  6. 2021Recent Advances in Natural Language Processing via Large Pre-Trained Language Models: A Surveycited by 457
  7. 2022RLPrompt: Optimizing Discrete Text Prompts with Reinforcement Learningcited by 361
  8. 2022Exploring Visual Prompts for Adapting Large-Scale Modelscited by 271
  9. 2024PiSSA: Principal Singular Values and Singular Vectors Adaptation of Large Language Modelscited by 261
  10. 2021Surface Form Competition: Why the Highest Probability Answer Isn't Always Rightcited by 256
  11. 2022Black-Box Tuning for Language-Model-as-a-Servicecited by 254
  12. 2023Think before you speak: Training Language Models With Pause Tokenscited by 254
38 more
  1. 2021SPoT: Better Frozen Model Adaptation through Soft Prompt Transfercited by 235
  2. 2021PPT: Pre-trained Prompt Tuning for Few-shot Learningcited by 232
  3. 2023Parameter-Efficient Fine-Tuning Methods for Pretrained Language Models: A Critical Review and Assessmentcited by 227
  4. 2022Surgical Fine-Tuning Improves Adaptation to Distribution Shiftscited by 205
  5. 2021Knowledgeable Prompt-tuning: Incorporating Knowledge into Prompt Verbalizer for Text Classificationcited by 183
  6. 2021KnowPrompt: Knowledge-aware Prompt-tuning with Synergistic Optimization for Relation Extractioncited by 180
  7. 2023Scaling Down to Scale Up: A Guide to Parameter-Efficient Fine-Tuningcited by 171
  8. 2023Active Prompting with Chain-of-Thought for Large Language Modelscited by 154
  9. 2022Can Machines Help Us Answering Question 16 in Datasheets, and In Turn Reflecting on Inappropriate Content?cited by 150
  10. 2023Large Language Models for Code: Security Hardening and Adversarial Testingcited by 149
  11. 2022Prompting PaLM for Translation: Assessing Strategies and Performancecited by 144
  12. 2023Domain Specialization as the Key to Make Large Language Models Disruptive: A Comprehensive Surveycited by 134
  13. 2024The Revolution of Multimodal Large Language Models: A Surveycited by 134
  14. 2022PromptFL: Let Federated Participants Cooperatively Learn Prompts Instead of Models -- Federated Learning in Age of Foundation Modelcited by 133
  15. 2023Automatic Prompt Augmentation and Selection with Chain-of-Thought from Labeled Datacited by 119
  16. 2021Differentiable Prompt Makes Pre-trained Language Models Better Few-shot Learnerscited by 115
  17. 2024The Butterfly Effect of Altering Prompts: How Small Changes and Jailbreaks Affect Large Language Model Performancecited by 97
  18. 2021Voice2Series: Reprogramming Acoustic Models for Time Series Classificationcited by 91
  19. 2024LISA: Layerwise Importance Sampling for Memory-Efficient Large Language Model Fine-Tuningcited by 88
  20. 2022FedPrompt: Communication-Efficient and Privacy Preserving Prompt Tuning in Federated Learningcited by 86
  21. 2021Why Do Pretrained Language Models Help in Downstream Tasks? An Analysis of Head and Prompt Tuningcited by 82
  22. 2023Modular Deep Learningcited by 82
  23. 2021What Changes Can Large-scale Language Models Bring? Intensive Study on HyperCLOVA: Billions-scale Korean Generative Pretrained Transformerscited by 75
  24. 2021Cross-Attention is All You Need: Adapting Pretrained Transformers for Machine Translationcited by 74
  25. 2021On Transferability of Prompt Tuning for Natural Language Processingcited by 74
  26. 2023Generative Speech Recognition Error Correction with Large Language Models and Task-Activating Promptingcited by 71
  27. 2022Black-box Prompt Learning for Pre-trained Language Modelscited by 70
  28. 2022HyperPrompt: Prompt-based Task-Conditioning of Transformerscited by 67
  29. 2022Model Reprogramming: Resource-Efficient Cross-Domain Machine Learningcited by 64
  30. 2023AntifakePrompt: Prompt-Tuned Vision-Language Models are Fake Image Detectorscited by 64
  31. 2023Exploring Diverse In-Context Configurations for Image Captioningcited by 63
  32. 2023Improving Visual Prompt Tuning for Self-supervised Vision Transformerscited by 57
  33. 2022Prototypical Verbalizer for Prompt-based Few-shot Tuningcited by 53
  34. 2023Fairness-guided Few-shot Prompting for Large Language Modelscited by 53
  35. 2021CPM-2: Large-scale Cost-effective Pre-trained Language Modelscited by 52
  36. 2023Fighting Fire with Fire: The Dual Role of LLMs in Crafting and Detecting Elusive Disinformationcited by 51
  37. 2022Toward Human Readable Prompt Tuning: Kubrick's The Shining is a good movie, and a good prompt too?cited by 49
  38. 2021Structural Adapters in Pretrained Language Models for AMR-to-text Generationcited by 48

50 most cited of 220 All 220 (JSON)