mlsanityData to 7 Oct 2026

Discrete and Soft Prompting for Multilingual Models

arXiv 2109.03630 ↗First posted 8 Sep 2021cs.CLcited by 49

Not on the map. Why?

Abstract (arXiv)

It has been shown for English that discrete and soft prompting perform strongly in few-shot learning with pretrained language models (PLMs). In this paper, we show that discrete and soft prompting perform better than finetuning in multilingual cases: Crosslingual transfer and in-language training of multilingual natural language inference. For example, with 48 English training examples, finetuning obtains 33.74% accuracy in crosslingual transfer, barely surpassing the majority baseline (33.33%). In contrast, discrete and soft prompting outperform finetuning, achieving 36.43% and 38.79%. We also demonstrate good performance of prompting with training data in multiple languages other than English.

Key facts. Cited by 49 papers in the map (data through 7 Oct 2026). Builds on 35 of its 41 references that are in the map. First posted on arXiv 8 Sep 2021.

Builds on

35 of 41 references.

References · 35

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
  4. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  5. 2019BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehensioncited by 5,874
  6. 2019Parameter-Efficient Transfer Learning for NLPcited by 4,195
  7. 2019Unsupervised Cross-lingual Representation Learning at Scalecited by 3,907
  8. 2021Prefix-Tuning: Optimizing Continuous Prompts for Generationcited by 3,878
  9. 2021The Power of Scale for Parameter-Efficient Prompt Tuningcited by 3,769
  10. 2019XLNet: Generalized Autoregressive Pretraining for Language Understandingcited by 2,801
  11. 2017A Broad-Coverage Challenge Corpus for Sentence Understanding through Inferencecited by 2,665
  12. 2019Language Models as Knowledge Bases?cited by 1,922
23 more
  1. 2015A large annotated corpus for learning natural language inferencecited by 1,837
  2. 2019SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systemscited by 1,718
  3. 2020Making Pre-trained Language Models Better Few-shot Learnerscited by 1,255
  4. 2018XNLI: Evaluating Cross-lingual Sentence Representationscited by 913
  5. 2020XTREME: A Massively Multilingual Multi-task Benchmark for Evaluating Cross-lingual Generalizationcited by 738
  6. 2021GPT Understands, Toocited by 715
  7. 2020It's Not Just Size That Matters: Small Language Models Are Also Few-Shot Learnerscited by 668
  8. 2019How multilingual is Multilingual BERT?cited by 642
  9. 2018Massively Multilingual Sentence Embeddings for Zero-Shot Cross-Lingual Transfer and Beyondcited by 434
  10. 2021Learning How to Ask: Querying LMs with Mixtures of Soft Promptscited by 376
  11. 2021True Few-Shot Learning with Language Modelscited by 343
  12. 2021Factual Probing Is [MASK]: Learning vs. Learning to Recallcited by 291
  13. 2020On the Stability of Fine-tuning BERT: Misconceptions, Explanations, and Strong Baselinescited by 254
  14. 2020Revisiting Few-sample BERT Fine-tuningcited by 237
  15. 2018The Natural Language Decathlon: Multitask Learning as Question Answeringcited by 235
  16. 2021How Many Data Points is a Prompt Worth?cited by 197
  17. 2021Improving and Simplifying Pattern Exploiting Trainingcited by 97
  18. 2020Masking as an Efficient Alternative to Finetuning for Pretrained Language Modelscited by 92
  19. 2021Few-Shot Question Answering by Pretraining Span Selectioncited by 62
  20. 2018Diverse Few-Shot Text Classification with Multiple Metricscited by 61
  21. 2020A Closer Look at Few-Shot Crosslingual Transfer: The Choice of Shots Matterscited by 45
  22. 2020Universal Natural Language Processing with Limited Annotations: Try Few-shot Textual Entailment as a Startcited by 44
  23. 2019Towards Realistic Practices In Low-Resource Natural Language Processing: The Development Setcited by 16

Built on by

Cited by year

Citing papers per year: 2021: 2; 2022: 18; 2023: 19; 2024: 8; 2025: 2.

49 papers.

Sort:

By area:

  1. 2022Language Models are Multilingual Chain-of-Thought Reasonerscited by 518
  2. 2023MEGA: Multilingual Evaluation of Generative AIcited by 291
  3. 2021Few-shot Learning with Multilingual Language Modelscited by 278
  4. 2022BLOOM+1: Adding Language Support to BLOOM for Zero-Shot Promptingcited by 90
  5. 2022Overcoming Catastrophic Forgetting in Zero-Shot Cross-Lingual Generationcited by 62
  6. 2022Modular and Parameter-Efficient Multimodal Fusion with Promptingcited by 35
  7. 2022Zero-Shot Rumor Detection with Propagation Structure via Prompt Learningcited by 32
  8. 2022Zero-shot Cross-lingual Transfer of Prompt-based Tuning with a Unified Multilingual Promptcited by 25
  9. 2022QAmeleon: Multilingual QA with Only 5 Examplescited by 25
  10. 2023RadAdapt: Radiology Report Summarization via Lightweight Domain Adaptation of Large Language Modelscited by 24
  11. 2022TyDiP: A Dataset for Politeness Classification in Nine Typologically Diverse Languagescited by 21
  12. 2022Towards Unified Prompt Tuning for Few-shot Text Classificationcited by 18
37 more
  1. 2022Cross-Lingual Retrieval Augmented Prompt for Low-Resource Languagescited by 17
  2. 2023From Classification to Generation: Insights into Crosslingual Retrieval Augmented ICLcited by 17
  3. 2022Prompt-Tuning Can Be Much Better Than Fine-Tuning on Cross-lingual Understanding With Multilingual Language Modelscited by 16
  4. 2022MULTI3NLU++: A Multilingual, Multi-Intent, Multi-Domain Dataset for Natural Language Understanding in Task-Oriented Dialoguecited by 15
  5. 2023Adapting an ASR Foundation Model for Spoken Language Assessmentcited by 14
  6. 2024PERSOMA: PERsonalized SOft ProMpt Adapter Architecture for Personalized Language Promptingcited by 13
  7. 2022HyperPELT: Unified Parameter-Efficient Language Model Tuning for Both Language and Vision-and-Language Taskscited by 12
  8. 2023In-Contextual Gender Bias Suppression for Large Language Modelscited by 12
  9. 2023Zero-shot Approach to Overcome Perturbation Sensitivity of Promptscited by 11
  10. 2022Shapley Head Pruning: Identifying and Removing Interference in Multilingual Transformerscited by 9
  11. 2023RAMP: Retrieval and Attribute-Marking Enhanced Prompting for Attribute-Controlled Translationcited by 9
  12. 2024Exploring the Limitations of Large Language Models in Compositional Relation Reasoningcited by 9
  13. 2021LMTurk: Few-Shot Learners as Crowdsourcing Workers in a Language-Model-as-a-Service Frameworkcited by 8
  14. 2023How Good are Commercial Large Language Models on African Languages?cited by 8
  15. 2023Bridging the Language Gap: Dynamic Learning Strategies for Improving Multilingual Performance in LLMscited by 8
  16. 2022Multilingual Relation Classification via Efficient and Effective Promptingcited by 7
  17. 2023Enhancing Cross-lingual Natural Language Inference by Soft Prompting with Multilingual Verbalizercited by 7
  18. 2023Is Prompt-Based Finetuning Always Better than Vanilla Finetuning? Insights from Cross-Lingual Language Understandingcited by 7
  19. 2023Crosslingual Retrieval Augmented In-context Learning for Banglacited by 7
  20. 2024ToPro: Token-Level Prompt Decomposition for Cross-Lingual Sequence Labeling Taskscited by 7
  21. 2024English Prompts are Better for NLI-based Zero-Shot Emotion Classification than Target-Language Promptscited by 7
  22. 2022Polyglot Prompt: Multilingual Multitask PrompTrainingcited by 6
  23. 2025Adapting Whisper for Parameter-efficient Code-Switching Speech Recognition via Soft Prompt Tuningcited by 5
  24. 2023Soft Language Clustering for Multilingual Model Pre-trainingcited by 4
  25. 2022Enhancing Cross-lingual Prompting with Dual Prompt Augmentationcited by 3
  26. 2024Cross-Lingual Transfer for Natural Language Inference via Multilingual Prompt Translatorcited by 3
  27. 2023Unleashing the Multilingual Encoder Potential: Boosting Zero-Shot Performance via Probability Calibrationcited by 2
  28. 2022This joke is [MASK]: Recognizing Humor and Offense with Promptingcited by 1
  29. 2023Towards Zero-Shot Functional Compositionality of Language Modelscited by 1
  30. 2023Sociocultural knowledge is needed for selection of shots in hate speech detection taskscited by 1
  31. 2024Few-Shot Cross-Lingual Transfer for Prompting Large Language Models in Low-Resource Languagescited by 1
  32. 2024Bridging the Language Gap: Enhancing Multilingual Prompt-Based Code Generation in LLMs via Zero-Shot Cross-Lingual Transfercited by 1
  33. 2025Statement-Tuning Enables Efficient Cross-lingual Generalization in Encoder-only Modelscited by 1
  34. 2023How to Solve Few-Shot Abusive Content Detection Using the Data We Actually Havecited by 0
  35. 2023Politeness Stereotypes and Attack Vectors: Gender Stereotypes in Japanese and Korean Language Modelscited by 0
  36. 2023TransPrompt v2: A Transferable Prompting Framework for Cross-task Text Classificationcited by 0
  37. 2024Effectively Prompting Small-sized Language Models for Cross-lingual Tasks via Winning Ticketscited by 0