# Is Prompt-Based Finetuning Always Better than Vanilla Finetuning? Insights from Cross-Lingual Language Understanding

arXiv: https://arxiv.org/abs/2307.07880 · HTML: https://mlsanity.com/paper/2307.07880/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- Builds on 37 of its 50 references that are in the map.
- First posted on arXiv 15 Jul 2023.

## Builds on (most cited first, 30 of 37)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Unsupervised Cross-lingual Representation Learning at Scale](https://mlsanity.com/paper/1911.02116/) (2019)
- [Prefix-Tuning: Optimizing Continuous Prompts for Generation](https://mlsanity.com/paper/2101.00190/) (2021)
- [XLNet: Generalized Autoregressive Pretraining for Language Understanding](https://mlsanity.com/paper/1906.08237/) (2019)
- [A Broad-Coverage Challenge Corpus for Sentence Understanding through Inference](https://mlsanity.com/paper/1704.05426/) (2017)
- [Making Pre-trained Language Models Better Few-shot Learners](https://mlsanity.com/paper/2012.15723/) (2020)
- [PEGASUS: Pre-training with Extracted Gap-sentences for Abstractive Summarization](https://mlsanity.com/paper/1912.08777/) (2019)
- [Exploiting Cloze Questions for Few Shot Text Classification and Natural Language Inference](https://mlsanity.com/paper/2001.07676/) (2020)
- [XNLI: Evaluating Cross-lingual Sentence Representations](https://mlsanity.com/paper/1809.05053/) (2018)
- [XTREME: A Massively Multilingual Multi-task Benchmark for Evaluating Cross-lingual Generalization](https://mlsanity.com/paper/2003.11080/) (2020)
- [It's Not Just Size That Matters: Small Language Models Are Also Few-Shot Learners](https://mlsanity.com/paper/2009.07118/) (2020)
- [How multilingual is Multilingual BERT?](https://mlsanity.com/paper/1906.01502/) (2019)
- [PAWS: Paraphrase Adversaries from Word Scrambling](https://mlsanity.com/paper/1904.01130/) (2019)
- [Learning How to Ask: Querying LMs with Mixtures of Soft Prompts](https://mlsanity.com/paper/2104.06599/) (2021)
- [Beto, Bentz, Becas: The Surprising Cross-Lingual Effectiveness of BERT](https://mlsanity.com/paper/1904.09077/) (2019)
- [PAWS-X: A Cross-lingual Adversarial Dataset for Paraphrase Identification](https://mlsanity.com/paper/1908.11828/) (2019)
- [XGLUE: A New Benchmark Dataset for Cross-lingual Pre-training, Understanding and Generation](https://mlsanity.com/paper/2004.01401/) (2020)
- [Cross-Lingual Ability of Multilingual BERT: An Empirical Study](https://mlsanity.com/paper/1912.07840/) (2019)
- [XTREME-R: Towards More Challenging and Nuanced Multilingual Evaluation](https://mlsanity.com/paper/2104.07412/) (2021)
- [The Multilingual Amazon Reviews Corpus](https://mlsanity.com/paper/2010.02573/) (2020)
- [Glot500: Scaling Multilingual Corpora and Language Models to 500 Languages](https://mlsanity.com/paper/2305.12182/) (2023)
- [Automatically Identifying Words That Can Serve as Labels for Few-Shot Text Classification](https://mlsanity.com/paper/2010.13641/) (2020)
- [Few-Shot Text Generation with Pattern-Exploiting Training](https://mlsanity.com/paper/2012.11926/) (2020)
- [Discrete and Soft Prompting for Multilingual Models](https://mlsanity.com/paper/2109.03630/) (2021)
- [Revisiting the Primacy of English in Zero-shot Cross-lingual Transfer](https://mlsanity.com/paper/2106.16171/) (2021)
- [Zero-shot Cross-lingual Transfer of Prompt-based Tuning with a Unified Multilingual Prompt](https://mlsanity.com/paper/2202.11451/) (2022)
- [Don't Stop Pretraining? Make Prompt-based Fine-tuning Powerful Learner](https://mlsanity.com/paper/2305.01711/) (2023)
- [Taxi1500: A Multilingual Dataset for Text Classification in 1500 Languages](https://mlsanity.com/paper/2305.08487/) (2023)

## Built on by (most cited first, 7 of 7)

- [AdaMergeX: Cross-Lingual Transfer with Large Language Models via Adaptive Adapter Merging](https://mlsanity.com/paper/2402.18913/) (2024)
- [ToPro: Token-Level Prompt Decomposition for Cross-Lingual Sequence Labeling Tasks](https://mlsanity.com/paper/2401.16589/) (2024)
- [Your Pretrained Model Tells the Difficulty Itself: A Self-Adaptive Curriculum Learning Paradigm for Natural Language Understanding](https://mlsanity.com/paper/2507.09758/) (2025)
- [Why Lift so Heavy? Slimming Large Language Models by Cutting Off the Layers](https://mlsanity.com/paper/2402.11700/) (2024)
- [Decomposed Prompting: Probing Multilingual Linguistic Structure Knowledge in Large Language Models](https://mlsanity.com/paper/2402.18397/) (2024)
- [GNNavi: Navigating the Information Flow in Large Language Models by Graph Neural Network](https://mlsanity.com/paper/2402.11709/) (2024)
- [Statement-Tuning Enables Efficient Cross-lingual Generalization in Encoder-only Models](https://mlsanity.com/paper/2506.01592/) (2025)

## Abstract (arXiv)

Multilingual pretrained language models (MPLMs) have demonstrated substantial performance improvements in zero-shot cross-lingual transfer across various natural language understanding tasks by finetuning MPLMs on task-specific labelled data of a source language (e.g. English) and evaluating on a wide range of target languages. Recent studies show that prompt-based finetuning surpasses regular finetuning in few-shot scenarios. However, the exploration of prompt-based learning in multilingual tasks remains limited. In this study, we propose the ProFiT pipeline to investigate the cross-lingual capabilities of Prompt-based Finetuning. We conduct comprehensive experiments on diverse cross-lingual language understanding tasks (sentiment classification, paraphrase identification, and natural language inference) and empirically analyze the variation trends of prompt-based finetuning performance in cross-lingual transfer across different few-shot and full-data settings. Our results reveal the effectiveness and versatility of prompt-based finetuning in cross-lingual language understanding. Our findings indicate that prompt-based finetuning outperforms vanilla finetuning in full-data scenarios and exhibits greater advantages in few-shot scenarios, with different performance patterns dependent on task types. Additionally, we analyze underlying factors such as language similarity and pretraining data size that impact the cross-lingual performance of prompt-based finetuning. Overall, our work provides valuable insights into the cross-lingual prowess of prompt-based finetuning.