# SwitchPrompt: Learning Domain-Specific Gated Soft Prompts for Classification in Low-Resource Domains

arXiv: https://arxiv.org/abs/2302.06868 · HTML: https://mlsanity.com/paper/2302.06868/

## Key facts

- Cited by 12 papers in the map (data through 14 Oct 2026).
- Builds on 22 of its 27 references that are in the map.
- First posted on arXiv 14 Feb 2023.

## Builds on (most cited first, 22 of 22)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [Prefix-Tuning: Optimizing Continuous Prompts for Generation](https://mlsanity.com/paper/2101.00190/) (2021)
- [The Power of Scale for Parameter-Efficient Prompt Tuning](https://mlsanity.com/paper/2104.08691/) (2021)
- [BioBERT: a pre-trained biomedical language representation model for biomedical text mining](https://mlsanity.com/paper/1901.08746/) (2019)
- [Don't Stop Pretraining: Adapt Language Models to Domains and Tasks](https://mlsanity.com/paper/2004.10964/) (2020)
- [SciBERT: A Pretrained Language Model for Scientific Text](https://mlsanity.com/paper/1903.10676/) (2019)
- [Making Pre-trained Language Models Better Few-shot Learners](https://mlsanity.com/paper/2012.15723/) (2020)
- [Aligning Books and Movies: Towards Story-like Visual Explanations by Watching Movies and Reading Books](https://mlsanity.com/paper/1506.06724/) (2015)
- [Publicly Available Clinical BERT Embeddings](https://mlsanity.com/paper/1904.03323/) (2019)
- [How Can We Know What Language Models Know?](https://mlsanity.com/paper/1911.12543/) (2019)
- [GPT Understands, Too](https://mlsanity.com/paper/2103.10385/) (2021)
- [Learning How to Ask: Querying LMs with Mixtures of Soft Prompts](https://mlsanity.com/paper/2104.06599/) (2021)
- [True Few-Shot Learning with Language Models](https://mlsanity.com/paper/2105.11447/) (2021)
- [A Survey on Recent Approaches for Natural Language Processing in Low-Resource Scenarios](https://mlsanity.com/paper/2010.12309/) (2020)
- [Larger-Scale Transformers for Multilingual Masked Language Modeling](https://mlsanity.com/paper/2105.00572/) (2021)
- [The SOFC-Exp Corpus and Neural Approaches to Information Extraction in the Materials Science Domain](https://mlsanity.com/paper/2006.03039/) (2020)
- [The Power of Prompt Tuning for Low-Resource Semantic Parsing](https://mlsanity.com/paper/2110.08525/) (2021)
- [Towards Realistic Practices In Low-Resource Natural Language Processing: The Development Set](https://mlsanity.com/paper/1909.01522/) (2019)
- [DomBERT: Domain-oriented Language Model for Aspect-based Sentiment Analysis](https://mlsanity.com/paper/2004.13816/) (2020)
- [CLIN-X: pre-trained language models and a study on cross-task transfer for concept extraction in the clinical domain](https://mlsanity.com/paper/2112.08754/) (2021)

## Built on by (most cited first, 12 of 12)

- [LLaMA-Adapter: Efficient Fine-tuning of Language Models with Zero-init Attention](https://mlsanity.com/paper/2303.16199/) (2023)
- [HiPerRAG: High-Performance Retrieval Augmented Generation for Scientific Insights](https://mlsanity.com/paper/2505.04846/) (2025)
- [CoPL: Contextual Prompt Learning for Vision-Language Understanding](https://mlsanity.com/paper/2307.00910/) (2023)
- [The Few-shot Dilemma: Over-prompting Large Language Models](https://mlsanity.com/paper/2509.13196/) (2025)
- [TADA: Efficient Task-Agnostic Domain Adaptation for Transformers](https://mlsanity.com/paper/2305.12717/) (2023)
- [Towards Automating Text Annotation: A Case Study on Semantic Proximity Annotation using GPT-4](https://mlsanity.com/paper/2407.04130/) (2024)
- [Fine-tuning Large Language Models with Limited Data: A Survey and Practical Guide](https://mlsanity.com/paper/2411.09539/) (2024)
- [Can Out-of-Domain data help to Learn Domain-Specific Prompts for Multimodal Misinformation Detection?](https://mlsanity.com/paper/2311.16496/) (2023)
- [Adaptive Query Prompting for Multi-Domain Landmark Detection](https://mlsanity.com/paper/2404.01194/) (2024)
- [SETTP: Style Extraction and Tunable Inference via Dual-level Transferable Prompt Learning](https://mlsanity.com/paper/2407.15556/) (2024)
- [Subjective Behaviors and Preferences in LLM: Language of Browsing](https://mlsanity.com/paper/2508.15474/) (2025)
- [Behind the [MASK]: Disentangling Representation and Faithfulness in DAPF-Based Dementia Detection](https://mlsanity.com/paper/2608.25028/) (2026)

## Abstract (arXiv)

Prompting pre-trained language models leads to promising results across natural language processing tasks but is less effective when applied in low-resource domains, due to the domain gap between the pre-training data and the downstream task. In this work, we bridge this gap with a novel and lightweight prompting methodology called SwitchPrompt for the adaptation of language models trained on datasets from the general domain to diverse low-resource domains. Using domain-specific keywords with a trainable gated prompt, SwitchPrompt offers domain-oriented prompting, that is, effective guidance on the target domains for general-domain language models. Our few-shot experiments on three text classification benchmarks demonstrate the efficacy of the general-domain pre-trained language models when used with SwitchPrompt. They often even outperform their domain-specific counterparts trained with baseline state-of-the-art prompting methods by up to 10.7% performance increase in accuracy. This result indicates that SwitchPrompt effectively reduces the need for domain-specific language model pre-training.