# Discrete and Soft Prompting for Multilingual Models

arXiv: https://arxiv.org/abs/2109.03630 · HTML: https://mlsanity.com/paper/2109.03630/

## Key facts

- Cited by 49 papers in the map (data through 14 Oct 2026).
- Builds on 35 of its 41 references that are in the map.
- First posted on arXiv 8 Sep 2021.

## Builds on (most cited first, 30 of 35)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension](https://mlsanity.com/paper/1910.13461/) (2019)
- [Parameter-Efficient Transfer Learning for NLP](https://mlsanity.com/paper/1902.00751/) (2019)
- [Unsupervised Cross-lingual Representation Learning at Scale](https://mlsanity.com/paper/1911.02116/) (2019)
- [Prefix-Tuning: Optimizing Continuous Prompts for Generation](https://mlsanity.com/paper/2101.00190/) (2021)
- [The Power of Scale for Parameter-Efficient Prompt Tuning](https://mlsanity.com/paper/2104.08691/) (2021)
- [XLNet: Generalized Autoregressive Pretraining for Language Understanding](https://mlsanity.com/paper/1906.08237/) (2019)
- [A Broad-Coverage Challenge Corpus for Sentence Understanding through Inference](https://mlsanity.com/paper/1704.05426/) (2017)
- [Language Models as Knowledge Bases?](https://mlsanity.com/paper/1909.01066/) (2019)
- [A large annotated corpus for learning natural language inference](https://mlsanity.com/paper/1508.05326/) (2015)
- [SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systems](https://mlsanity.com/paper/1905.00537/) (2019)
- [Making Pre-trained Language Models Better Few-shot Learners](https://mlsanity.com/paper/2012.15723/) (2020)
- [XNLI: Evaluating Cross-lingual Sentence Representations](https://mlsanity.com/paper/1809.05053/) (2018)
- [XTREME: A Massively Multilingual Multi-task Benchmark for Evaluating Cross-lingual Generalization](https://mlsanity.com/paper/2003.11080/) (2020)
- [GPT Understands, Too](https://mlsanity.com/paper/2103.10385/) (2021)
- [It's Not Just Size That Matters: Small Language Models Are Also Few-Shot Learners](https://mlsanity.com/paper/2009.07118/) (2020)
- [How multilingual is Multilingual BERT?](https://mlsanity.com/paper/1906.01502/) (2019)
- [Massively Multilingual Sentence Embeddings for Zero-Shot Cross-Lingual Transfer and Beyond](https://mlsanity.com/paper/1812.10464/) (2018)
- [Learning How to Ask: Querying LMs with Mixtures of Soft Prompts](https://mlsanity.com/paper/2104.06599/) (2021)
- [True Few-Shot Learning with Language Models](https://mlsanity.com/paper/2105.11447/) (2021)
- [Factual Probing Is [MASK]: Learning vs. Learning to Recall](https://mlsanity.com/paper/2104.05240/) (2021)
- [On the Stability of Fine-tuning BERT: Misconceptions, Explanations, and Strong Baselines](https://mlsanity.com/paper/2006.04884/) (2020)
- [Revisiting Few-sample BERT Fine-tuning](https://mlsanity.com/paper/2006.05987/) (2020)
- [The Natural Language Decathlon: Multitask Learning as Question Answering](https://mlsanity.com/paper/1806.08730/) (2018)
- [How Many Data Points is a Prompt Worth?](https://mlsanity.com/paper/2103.08493/) (2021)
- [Improving and Simplifying Pattern Exploiting Training](https://mlsanity.com/paper/2103.11955/) (2021)
- [Masking as an Efficient Alternative to Finetuning for Pretrained Language Models](https://mlsanity.com/paper/2004.12406/) (2020)

## Built on by (most cited first, 30 of 49)

- [Language Models are Multilingual Chain-of-Thought Reasoners](https://mlsanity.com/paper/2210.03057/) (2022)
- [MEGA: Multilingual Evaluation of Generative AI](https://mlsanity.com/paper/2303.12528/) (2023)
- [Few-shot Learning with Multilingual Language Models](https://mlsanity.com/paper/2112.10668/) (2021)
- [BLOOM+1: Adding Language Support to BLOOM for Zero-Shot Prompting](https://mlsanity.com/paper/2212.09535/) (2022)
- [Overcoming Catastrophic Forgetting in Zero-Shot Cross-Lingual Generation](https://mlsanity.com/paper/2205.12647/) (2022)
- [Modular and Parameter-Efficient Multimodal Fusion with Prompting](https://mlsanity.com/paper/2203.08055/) (2022)
- [Zero-Shot Rumor Detection with Propagation Structure via Prompt Learning](https://mlsanity.com/paper/2212.01117/) (2022)
- [Zero-shot Cross-lingual Transfer of Prompt-based Tuning with a Unified Multilingual Prompt](https://mlsanity.com/paper/2202.11451/) (2022)
- [QAmeleon: Multilingual QA with Only 5 Examples](https://mlsanity.com/paper/2211.08264/) (2022)
- [RadAdapt: Radiology Report Summarization via Lightweight Domain Adaptation of Large Language Models](https://mlsanity.com/paper/2305.01146/) (2023)
- [TyDiP: A Dataset for Politeness Classification in Nine Typologically Diverse Languages](https://mlsanity.com/paper/2211.16496/) (2022)
- [Towards Unified Prompt Tuning for Few-shot Text Classification](https://mlsanity.com/paper/2205.05313/) (2022)
- [Cross-Lingual Retrieval Augmented Prompt for Low-Resource Languages](https://mlsanity.com/paper/2212.09651/) (2022)
- [From Classification to Generation: Insights into Crosslingual Retrieval Augmented ICL](https://mlsanity.com/paper/2311.06595/) (2023)
- [Prompt-Tuning Can Be Much Better Than Fine-Tuning on Cross-lingual Understanding With Multilingual Language Models](https://mlsanity.com/paper/2210.12360/) (2022)
- [MULTI3NLU++: A Multilingual, Multi-Intent, Multi-Domain Dataset for Natural Language Understanding in Task-Oriented Dialogue](https://mlsanity.com/paper/2212.10455/) (2022)
- [Adapting an ASR Foundation Model for Spoken Language Assessment](https://mlsanity.com/paper/2307.09378/) (2023)
- [PERSOMA: PERsonalized SOft ProMpt Adapter Architecture for Personalized Language Prompting](https://mlsanity.com/paper/2408.00960/) (2024)
- [HyperPELT: Unified Parameter-Efficient Language Model Tuning for Both Language and Vision-and-Language Tasks](https://mlsanity.com/paper/2203.03878/) (2022)
- [In-Contextual Gender Bias Suppression for Large Language Models](https://mlsanity.com/paper/2309.07251/) (2023)
- [Zero-shot Approach to Overcome Perturbation Sensitivity of Prompts](https://mlsanity.com/paper/2305.15689/) (2023)
- [LMTurk: Few-Shot Learners as Crowdsourcing Workers in a Language-Model-as-a-Service Framework](https://mlsanity.com/paper/2112.07522/) (2021)
- [Shapley Head Pruning: Identifying and Removing Interference in Multilingual Transformers](https://mlsanity.com/paper/2210.05709/) (2022)
- [RAMP: Retrieval and Attribute-Marking Enhanced Prompting for Attribute-Controlled Translation](https://mlsanity.com/paper/2305.17131/) (2023)
- [Exploring the Limitations of Large Language Models in Compositional Relation Reasoning](https://mlsanity.com/paper/2403.02615/) (2024)
- [How Good are Commercial Large Language Models on African Languages?](https://mlsanity.com/paper/2305.06530/) (2023)
- [Bridging the Language Gap: Dynamic Learning Strategies for Improving Multilingual Performance in LLMs](https://mlsanity.com/paper/2305.17740/) (2023)
- [Multilingual Relation Classification via Efficient and Effective Prompting](https://mlsanity.com/paper/2210.13838/) (2022)
- [Enhancing Cross-lingual Natural Language Inference by Soft Prompting with Multilingual Verbalizer](https://mlsanity.com/paper/2305.12761/) (2023)
- [Is Prompt-Based Finetuning Always Better than Vanilla Finetuning? Insights from Cross-Lingual Language Understanding](https://mlsanity.com/paper/2307.07880/) (2023)

## Abstract (arXiv)

It has been shown for English that discrete and soft prompting perform strongly in few-shot learning with pretrained language models (PLMs). In this paper, we show that discrete and soft prompting perform better than finetuning in multilingual cases: Crosslingual transfer and in-language training of multilingual natural language inference. For example, with 48 English training examples, finetuning obtains 33.74% accuracy in crosslingual transfer, barely surpassing the majority baseline (33.33%). In contrast, discrete and soft prompting outperform finetuning, achieving 36.43% and 38.79%. We also demonstrate good performance of prompting with training data in multiple languages other than English.