# Zero-shot Cross-lingual Transfer of Prompt-based Tuning with a Unified Multilingual Prompt

arXiv: https://arxiv.org/abs/2202.11451 · HTML: https://mlsanity.com/paper/2202.11451/

## Key facts

- Cited by 25 papers in the map (data through 14 Oct 2026).
- Builds on 18 of its 21 references that are in the map.
- First posted on arXiv 23 Feb 2022.

## Builds on (most cited first, 18 of 18)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [Unsupervised Cross-lingual Representation Learning at Scale](https://mlsanity.com/paper/1911.02116/) (2019)
- [Prefix-Tuning: Optimizing Continuous Prompts for Generation](https://mlsanity.com/paper/2101.00190/) (2021)
- [XLNet: Generalized Autoregressive Pretraining for Language Understanding](https://mlsanity.com/paper/1906.08237/) (2019)
- [Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processing](https://mlsanity.com/paper/2107.13586/) (2021)
- [Making Pre-trained Language Models Better Few-shot Learners](https://mlsanity.com/paper/2012.15723/) (2020)
- [P-Tuning v2: Prompt Tuning Can Be Comparable to Fine-tuning Universally Across Scales and Tasks](https://mlsanity.com/paper/2110.07602/) (2021)
- [WARP: Word-level Adversarial ReProgramming](https://mlsanity.com/paper/2101.00121/) (2021)
- [How Many Data Points is a Prompt Worth?](https://mlsanity.com/paper/2103.08493/) (2021)
- [SimAlign: High Quality Word Alignments without Parallel Training Data using Static and Contextualized Embeddings](https://mlsanity.com/paper/2004.08728/) (2020)
- [The Multilingual Amazon Reviews Corpus](https://mlsanity.com/paper/2010.02573/) (2020)
- [Intermediate-Task Transfer Learning with Pretrained Models for Natural Language Understanding: When and Why Does It Work?](https://mlsanity.com/paper/2005.00628/) (2020)
- [Differentiable Prompt Makes Pre-trained Language Models Better Few-shot Learners](https://mlsanity.com/paper/2108.13161/) (2021)
- [Language Models are Few-shot Multilingual Learners](https://mlsanity.com/paper/2109.07684/) (2021)
- [Discrete and Soft Prompting for Multilingual Models](https://mlsanity.com/paper/2109.03630/) (2021)
- [A Closer Look at Few-Shot Crosslingual Transfer: The Choice of Shots Matters](https://mlsanity.com/paper/2012.15682/) (2020)

## Built on by (most cited first, 25 of 25)

- [Enhancing Multilingual Capabilities of Large Language Models through Self-Distillation from Resource-Rich Languages](https://mlsanity.com/paper/2402.12204/) (2024)
- [Zero-Shot Rumor Detection with Propagation Structure via Prompt Learning](https://mlsanity.com/paper/2212.01117/) (2022)
- [Is Translation All You Need? A Study on Solving Multilingual Tasks with Large Language Models](https://mlsanity.com/paper/2403.10258/) (2024)
- [InfoPrompt: Information-Theoretic Soft Prompt Tuning for Natural Language Understanding](https://mlsanity.com/paper/2306.04933/) (2023)
- [Cross-Lingual Retrieval Augmented Prompt for Low-Resource Languages](https://mlsanity.com/paper/2212.09651/) (2022)
- [From Classification to Generation: Insights into Crosslingual Retrieval Augmented ICL](https://mlsanity.com/paper/2311.06595/) (2023)
- [Prompt-Tuning Can Be Much Better Than Fine-Tuning on Cross-lingual Understanding With Multilingual Language Models](https://mlsanity.com/paper/2210.12360/) (2022)
- [RAMP: Retrieval and Attribute-Marking Enhanced Prompting for Attribute-Controlled Translation](https://mlsanity.com/paper/2305.17131/) (2023)
- [Bridging the Language Gap: Dynamic Learning Strategies for Improving Multilingual Performance in LLMs](https://mlsanity.com/paper/2305.17740/) (2023)
- [Multilingual Relation Classification via Efficient and Effective Prompting](https://mlsanity.com/paper/2210.13838/) (2022)
- [Is Prompt-Based Finetuning Always Better than Vanilla Finetuning? Insights from Cross-Lingual Language Understanding](https://mlsanity.com/paper/2307.07880/) (2023)
- [Crosslingual Retrieval Augmented In-context Learning for Bangla](https://mlsanity.com/paper/2311.00587/) (2023)
- [ToPro: Token-Level Prompt Decomposition for Cross-Lingual Sequence Labeling Tasks](https://mlsanity.com/paper/2401.16589/) (2024)
- [English Prompts are Better for NLI-based Zero-Shot Emotion Classification than Target-Language Prompts](https://mlsanity.com/paper/2402.03223/) (2024)
- [Soft Prompt Tuning for Cross-Lingual Transfer: When Less is More](https://mlsanity.com/paper/2402.03782/) (2024)
- [Learning from Contrastive Prompts: Automated Optimization and Adaptation](https://mlsanity.com/paper/2409.15199/) (2024)
- [The Roles of English in Evaluating Multilingual Language Models](https://mlsanity.com/paper/2412.08392/) (2024)
- [Soft Language Prompts for Language Transfer](https://mlsanity.com/paper/2407.02317/) (2024)
- [Adapting Whisper for Parameter-efficient Code-Switching Speech Recognition via Soft Prompt Tuning](https://mlsanity.com/paper/2506.21576/) (2025)
- [Soft Language Clustering for Multilingual Model Pre-training](https://mlsanity.com/paper/2306.07610/) (2023)
- [Unleashing the Multilingual Encoder Potential: Boosting Zero-Shot Performance via Probability Calibration](https://mlsanity.com/paper/2310.05069/) (2023)
- [On the Analysis of Cross-Lingual Prompt Tuning for Decoder-based Multilingual Model](https://mlsanity.com/paper/2311.07820/) (2023)
- [Multilingual Few-Shot Learning via Language Model Retrieval](https://mlsanity.com/paper/2306.10964/) (2023)
- [Bridging the Language Gap: Enhancing Multilingual Prompt-Based Code Generation in LLMs via Zero-Shot Cross-Lingual Transfer](https://mlsanity.com/paper/2408.09701/) (2024)
- [Cross-Prompt Encoder for Low-Performing Languages](https://mlsanity.com/paper/2508.10352/) (2025)

## Abstract (arXiv)

Prompt-based tuning has been proven effective for pretrained language models (PLMs). While most of the existing work focuses on the monolingual prompts, we study the multilingual prompts for multilingual PLMs, especially in the zero-shot cross-lingual setting. To alleviate the effort of designing different prompts for multiple languages, we propose a novel model that uses a unified prompt for all languages, called UniPrompt. Different from the discrete prompts and soft prompts, the unified prompt is model-based and language-agnostic. Specifically, the unified prompt is initialized by a multilingual PLM to produce language-independent representation, after which is fused with the text input. During inference, the prompts can be pre-computed so that no extra computation cost is needed. To collocate with the unified prompt, we propose a new initialization method for the target label word to further improve the model's transferability across languages. Extensive experiments show that our proposed methods can significantly outperform the strong baselines across different languages. We release data and code to facilitate future research.