# Prompt-Tuning Can Be Much Better Than Fine-Tuning on Cross-lingual Understanding With Multilingual Language Models

arXiv: https://arxiv.org/abs/2210.12360 · HTML: https://mlsanity.com/paper/2210.12360/

## Key facts

- Cited by 16 papers in the map (data through 14 Oct 2026).
- Builds on 23 of its 31 references that are in the map.
- First posted on arXiv 22 Oct 2022.

## Builds on (most cited first, 23 of 23)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Unsupervised Cross-lingual Representation Learning at Scale](https://mlsanity.com/paper/1911.02116/) (2019)
- [Prefix-Tuning: Optimizing Continuous Prompts for Generation](https://mlsanity.com/paper/2101.00190/) (2021)
- [The Power of Scale for Parameter-Efficient Prompt Tuning](https://mlsanity.com/paper/2104.08691/) (2021)
- [Cross-lingual Language Model Pretraining](https://mlsanity.com/paper/1901.07291/) (2019)
- [XNLI: Evaluating Cross-lingual Sentence Representations](https://mlsanity.com/paper/1809.05053/) (2018)
- [XTREME: A Massively Multilingual Multi-task Benchmark for Evaluating Cross-lingual Generalization](https://mlsanity.com/paper/2003.11080/) (2020)
- [How multilingual is Multilingual BERT?](https://mlsanity.com/paper/1906.01502/) (2019)
- [On the Cross-lingual Transferability of Monolingual Representations](https://mlsanity.com/paper/1910.11856/) (2019)
- [ERNIE: Enhanced Language Representation with Informative Entities](https://mlsanity.com/paper/1905.07129/) (2019)
- [TyDi QA: A Benchmark for Information-Seeking Question Answering in Typologically Diverse Languages](https://mlsanity.com/paper/2003.05002/) (2020)
- [Beto, Bentz, Becas: The Surprising Cross-Lingual Effectiveness of BERT](https://mlsanity.com/paper/1904.09077/) (2019)
- [Factual Probing Is [MASK]: Learning vs. Learning to Recall](https://mlsanity.com/paper/2104.05240/) (2021)
- [PAWS-X: A Cross-lingual Adversarial Dataset for Paraphrase Identification](https://mlsanity.com/paper/1908.11828/) (2019)
- [XGLUE: A New Benchmark Dataset for Cross-lingual Pre-training, Understanding and Generation](https://mlsanity.com/paper/2004.01401/) (2020)
- [WARP: Word-level Adversarial ReProgramming](https://mlsanity.com/paper/2101.00121/) (2021)
- [XTREME-R: Towards More Challenging and Nuanced Multilingual Evaluation](https://mlsanity.com/paper/2104.07412/) (2021)
- [Language Models are Few-shot Multilingual Learners](https://mlsanity.com/paper/2109.07684/) (2021)
- [Discrete and Soft Prompting for Multilingual Models](https://mlsanity.com/paper/2109.03630/) (2021)
- [VECO: Variable and Flexible Cross-lingual Pre-training for Language Understanding and Generation](https://mlsanity.com/paper/2010.16046/) (2020)
- [Zero-shot Cross-lingual Transfer of Prompt-based Tuning with a Unified Multilingual Prompt](https://mlsanity.com/paper/2202.11451/) (2022)

## Built on by (most cited first, 16 of 16)

- [A Survey on RAG Meeting LLMs: Towards Retrieval-Augmented Large Language Models](https://mlsanity.com/paper/2405.06211/) (2024)
- [BLOOM+1: Adding Language Support to BLOOM for Zero-Shot Prompting](https://mlsanity.com/paper/2212.09535/) (2022)
- [Facing the Elephant in the Room: Visual Prompt Tuning or Full Finetuning?](https://mlsanity.com/paper/2401.12902/) (2024)
- [Enhancing Domain Adaptation through Prompt Gradient Alignment](https://mlsanity.com/paper/2406.09353/) (2024)
- [Is Prompt-Based Finetuning Always Better than Vanilla Finetuning? Insights from Cross-Lingual Language Understanding](https://mlsanity.com/paper/2307.07880/) (2023)
- [ToPro: Token-Level Prompt Decomposition for Cross-Lingual Sequence Labeling Tasks](https://mlsanity.com/paper/2401.16589/) (2024)
- [English Prompts are Better for NLI-based Zero-Shot Emotion Classification than Target-Language Prompts](https://mlsanity.com/paper/2402.03223/) (2024)
- [Efficiently Aligned Cross-Lingual Transfer Learning for Conversational Tasks using Prompt-Tuning](https://mlsanity.com/paper/2304.01295/) (2023)
- [Analysis of Multi-Source Language Training in Cross-Lingual Transfer](https://mlsanity.com/paper/2402.13562/) (2024)
- [Breaking the Language Barrier: Improving Cross-Lingual Reasoning with Structured Self-Attention](https://mlsanity.com/paper/2310.15258/) (2023)
- [UniBridge: A Unified Approach to Cross-Lingual Transfer Learning for Low-Resource Languages](https://mlsanity.com/paper/2406.09717/) (2024)
- [On the Analysis of Cross-Lingual Prompt Tuning for Decoder-based Multilingual Model](https://mlsanity.com/paper/2311.07820/) (2023)
- [Prompt Tuned Embedding Classification for Multi-Label Industry Sector Allocation](https://mlsanity.com/paper/2309.12075/) (2023)
- [Statement-Tuning Enables Efficient Cross-lingual Generalization in Encoder-only Models](https://mlsanity.com/paper/2506.01592/) (2025)
- [Prompt-based Personality Profiling: Reinforcement Learning for Relevance Filtering](https://mlsanity.com/paper/2409.04122/) (2024)
- [Zero-Shot Cross-Lingual Transfer using Prefix-Based Adaptation](https://mlsanity.com/paper/2510.24619/) (2025)

## Abstract (arXiv)

Pre-trained multilingual language models show significant performance gains for zero-shot cross-lingual model transfer on a wide range of natural language understanding (NLU) tasks. Previously, for zero-shot cross-lingual evaluation, pre-trained models are only fine-tuned on English data and tested on a variety of target languages. In this paper, we do cross-lingual evaluation on various NLU tasks (sentence classification, sequence labeling, question answering) using prompt-tuning and compare it with fine-tuning. The results show that prompt tuning achieves much better cross-lingual transfer than fine-tuning across datasets, with only 0.1% to 0.3% tuned parameters. Additionally, we demonstrate through the analysis that prompt tuning can have better cross-lingual transferability of representations on downstream tasks with better aligned decision boundaries.