# PADA: Example-based Prompt Learning for on-the-fly Adaptation to Unseen Domains

arXiv: https://arxiv.org/abs/2102.12206 · HTML: https://mlsanity.com/paper/2102.12206/

## Key facts

- Cited by 61 papers in the map (data through 14 Oct 2026).
- Builds on 36 of its 83 references that are in the map.
- First posted on arXiv 24 Feb 2021.

## Builds on (most cited first, 30 of 36)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension](https://mlsanity.com/paper/1910.13461/) (2019)
- [BERTScore: Evaluating Text Generation with BERT](https://mlsanity.com/paper/1904.09675/) (2019)
- [DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter](https://mlsanity.com/paper/1910.01108/) (2019)
- [Prefix-Tuning: Optimizing Continuous Prompts for Generation](https://mlsanity.com/paper/2101.00190/) (2021)
- [Domain-Adversarial Training of Neural Networks](https://mlsanity.com/paper/1505.07818/) (2015)
- [A Broad-Coverage Challenge Corpus for Sentence Understanding through Inference](https://mlsanity.com/paper/1704.05426/) (2017)
- [Invariant Risk Minimization](https://mlsanity.com/paper/1907.02893/) (2019)
- [A large annotated corpus for learning natural language inference](https://mlsanity.com/paper/1508.05326/) (2015)
- [WILDS: A Benchmark of in-the-Wild Distribution Shifts](https://mlsanity.com/paper/2012.07421/) (2020)
- [Making Pre-trained Language Models Better Few-shot Learners](https://mlsanity.com/paper/2012.15723/) (2020)
- [How Can We Know What Language Models Know?](https://mlsanity.com/paper/1911.12543/) (2019)
- [Exploiting Cloze Questions for Few Shot Text Classification and Natural Language Inference](https://mlsanity.com/paper/2001.07676/) (2020)
- [Domain Generalization via Invariant Feature Representation](https://mlsanity.com/paper/1301.2115/) (2013)
- [MAD-X: An Adapter-Based Framework for Multi-Task Cross-Lingual Transfer](https://mlsanity.com/paper/2005.00052/) (2020)
- [Diverse Beam Search: Decoding Diverse Solutions from Neural Sequence Models](https://mlsanity.com/paper/1610.02424/) (2016)
- [Continual learning with hypernetworks](https://mlsanity.com/paper/1906.00695/) (2019)
- [AutoPrompt: Eliciting Knowledge from Language Models with Automatically Generated Prompts](https://mlsanity.com/paper/2010.15980/) (2020)
- [Neural Unsupervised Domain Adaptation in NLP---A Survey](https://mlsanity.com/paper/2006.00632/) (2020)
- [Distributionally Robust Language Modeling](https://mlsanity.com/paper/1909.02060/) (2019)
- [On Calibration and Out-of-domain Generalization](https://mlsanity.com/paper/2102.10395/) (2021)
- [Domain Robustness in Neural Machine Translation](https://mlsanity.com/paper/1911.03109/) (2019)
- [Analysing How People Orient to and Spread Rumours in Social Media by Looking at Conversational Threads](https://mlsanity.com/paper/1511.07487/) (2015)
- [Multi-Source Domain Adaptation with Mixture of Experts](https://mlsanity.com/paper/1809.02256/) (2018)
- [Multinomial Adversarial Networks for Multi-Domain Text Classification](https://mlsanity.com/paper/1802.05694/) (2018)
- [Does Distributionally Robust Supervised Learning Give Robust Classifiers?](https://mlsanity.com/paper/1611.02041/) (2016)

## Built on by (most cited first, 30 of 61)

- [Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processing](https://mlsanity.com/paper/2107.13586/) (2021)
- [A Comprehensive Survey on Test-Time Adaptation under Distribution Shifts](https://mlsanity.com/paper/2303.15361/) (2023)
- [Domain Specialization as the Key to Make Large Language Models Disruptive: A Comprehensive Survey](https://mlsanity.com/paper/2305.18703/) (2023)
- [Automatic Prompt Augmentation and Selection with Chain-of-Thought from Labeled Data](https://mlsanity.com/paper/2302.12822/) (2023)
- [Personalized Prompt Learning for Explainable Recommendation](https://mlsanity.com/paper/2202.07371/) (2022)
- [Black-box Prompt Learning for Pre-trained Language Models](https://mlsanity.com/paper/2201.08531/) (2022)
- [PRewrite: Prompt Rewriting with Reinforcement Learning](https://mlsanity.com/paper/2401.08189/) (2024)
- [Faithful Explanations of Black-box NLP Models Using LLM-generated Counterfactuals](https://mlsanity.com/paper/2310.00603/) (2023)
- [SAMAug: Point Prompt Augmentation for Segment Anything Model](https://mlsanity.com/paper/2307.01187/) (2023)
- [LightNER: A Lightweight Tuning Paradigm for Low-resource NER via Pluggable Prompting](https://mlsanity.com/paper/2109.00720/) (2021)
- [ClickPrompt: CTR Models are Strong Prompt Generators for Adapting Language Models to CTR Prediction](https://mlsanity.com/paper/2310.09234/) (2023)
- [Multi-Prompt Alignment for Multi-Source Unsupervised Domain Adaptation](https://mlsanity.com/paper/2209.15210/) (2022)
- [Decoding Time Series with LLMs: A Multi-Agent Framework for Cross-Domain Annotation](https://mlsanity.com/paper/2410.17462/) (2024)
- [Prompt-Augmented Linear Probing: Scaling beyond the Limit of Few-shot In-Context Learners](https://mlsanity.com/paper/2212.10873/) (2022)
- [No Parameter Left Behind: How Distillation and Model Size Affect Zero-Shot Retrieval](https://mlsanity.com/paper/2206.02873/) (2022)
- [Example-based Hypernetworks for Out-of-Distribution Generalization](https://mlsanity.com/paper/2203.14276/) (2022)
- [AdaptiVocab: Enhancing LLM Efficiency in Focused Domains through Lightweight Vocabulary Adaptation](https://mlsanity.com/paper/2503.19693/) (2025)
- [Zero-shot Transfer of Article-aware Legal Outcome Classification for European Court of Human Rights Cases](https://mlsanity.com/paper/2302.00609/) (2023)
- [Zero-shot Domain-sensitive Speech Recognition with Prompt-conditioning Fine-tuning](https://mlsanity.com/paper/2307.10274/) (2023)
- [FlexiReg: Flexible Urban Region Representation Learning](https://mlsanity.com/paper/2503.09128/) (2025)
- [On the Robustness of Dialogue History Representation in Conversational Question Answering: A Comprehensive Study and a New Prompt-based Method](https://mlsanity.com/paper/2206.14796/) (2022)
- [Efficient Federated Prompt Tuning for Black-box Large Pre-trained Models](https://mlsanity.com/paper/2310.03123/) (2023)
- [Measuring the Robustness of NLP Models to Domain Shifts](https://mlsanity.com/paper/2306.00168/) (2023)
- [Prompt Valuation Based on Shapley Values](https://mlsanity.com/paper/2312.15395/) (2023)
- [A Sentence is Worth 128 Pseudo Tokens: A Semantic-Aware Contrastive Learning Framework for Sentence Embeddings](https://mlsanity.com/paper/2203.05877/) (2022)
- [Learning a Better Initialization for Soft Prompts via Meta-Learning](https://mlsanity.com/paper/2205.12471/) (2022)
- [Billions of Parameters Are Worth More Than In-domain Training Data: A case study in the Legal Case Entailment Task](https://mlsanity.com/paper/2205.15172/) (2022)
- [Robust Question Answering against Distribution Shifts with Test-Time Adaptation: An Empirical Study](https://mlsanity.com/paper/2302.04618/) (2023)
- [Plum: Prompt Learning using Metaheuristic](https://mlsanity.com/paper/2311.08364/) (2023)
- [Speaking the Language of Your Listener: Audience-Aware Adaptation via Plug-and-Play Theory of Mind](https://mlsanity.com/paper/2305.19933/) (2023)

## Abstract (arXiv)

Natural Language Processing algorithms have made incredible progress, but they still struggle when applied to out-of-distribution examples. We address a challenging and underexplored version of this domain adaptation problem, where an algorithm is trained on several source domains, and then applied to examples from unseen domains that are unknown at training time. Particularly, no examples, labeled or unlabeled, or any other knowledge about the target domain are available to the algorithm at training time. We present PADA: An example-based autoregressive Prompt learning algorithm for on-the-fly Any-Domain Adaptation, based on the T5 language model. Given a test example, PADA first generates a unique prompt for it and then, conditioned on this prompt, labels the example with respect to the NLP prediction task. PADA is trained to generate a prompt which is a token sequence of unrestricted length, consisting of Domain Related Features (DRFs) that characterize each of the source domains. Intuitively, the generated prompt is a unique signature that maps the test example to a semantic space spanned by the source domains. In experiments with 3 tasks (text classification and sequence tagging), for a total of 14 multi-source adaptation scenarios, PADA substantially outperforms strong baselines.