mlsanityData to 7 Oct 2026

MSP: Multi-Stage Prompting for Making Pre-trained Language Models Better Translators

arXiv 2110.06609 ↗First posted 13 Oct 2021cs.CLcited by 35

Not on the map. Why?

Abstract (arXiv)

Prompting has recently been shown as a promising approach for applying pre-trained language models to perform downstream tasks. We present Multi-Stage Prompting (MSP), a simple and automatic approach for leveraging pre-trained language models to translation tasks. To better mitigate the discrepancy between pre-training and translation, MSP divides the translation process via pre-trained language models into multiple separate stages: the encoding stage, the re-encoding stage, and the decoding stage. During each stage, we independently apply different continuous prompts for allowing pre-trained language models better shift to translation tasks. We conduct extensive experiments on three translation tasks. Experiments show that our method can significantly improve the translation performance of pre-trained language models.

Key facts. Cited by 35 papers in the map (data through 7 Oct 2026). Builds on 28 of its 37 references that are in the map. First posted on arXiv 13 Oct 2021.

Builds on

28 of 37 references.

References · 28

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  4. 2020Language Models are Few-Shot Learnerscited by 32,199
  5. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  6. 2014Neural Machine Translation by Jointly Learning to Align and Translatecited by 4,530
  7. 2021Prefix-Tuning: Optimizing Continuous Prompts for Generationcited by 3,878
  8. 2021The Power of Scale for Parameter-Efficient Prompt Tuningcited by 3,769
  9. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,747
  10. 2014Sequence to Sequence Learning with Neural Networkscited by 2,974
  11. 2021Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processingcited by 2,222
  12. 2019Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelismcited by 1,829
16 more
  1. 2020mT5: A massively multilingual pre-trained text-to-text transformercited by 1,813
  2. 2018A Call for Clarity in Reporting BLEU Scorescited by 1,790
  3. 2020Making Pre-trained Language Models Better Few-shot Learnerscited by 1,255
  4. 2021GPT Understands, Toocited by 715
  5. 2017Six Challenges for Neural Machine Translationcited by 334
  6. 2021ERNIE 3.0: Large-scale Knowledge Enhanced Pre-training for Language Understanding and Generationcited by 258
  7. 2019Simple, Scalable Adaptation for Neural Machine Translationcited by 239
  8. 2018When and Why are Pre-trained Word Embeddings Useful for Neural Machine Translation?cited by 125
  9. 2021CPM-2: Large-scale Cost-effective Pre-trained Language Modelscited by 52
  10. 2019Towards Making the Most of BERT in Neural Machine Translationcited by 43
  11. 2020Incorporating BERT into Parallel Sequence Decoding with Adapterscited by 39
  12. 2019Acquiring Knowledge from Pre-trained Model to Neural Machine Translationcited by 30
  13. 2020Recipes for Adapting Pre-trained Monolingual and Multilingual Models to Machine Translationcited by 28
  14. 2018Multilingual Neural Machine Translation with Task-Specific Attentioncited by 26
  15. 2021Language Models are Good Translatorscited by 16
  16. 2021Multilingual Translation via Grafting Pre-trained Language Modelscited by 10

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 1; 2022: 9; 2023: 9; 2024: 12; 2025: 3; 2026: 1.

35 papers.

Sort:

By area:

  1. 2022GPT-NeoX-20B: An Open-Source Autoregressive Language Modelcited by 711
  2. 2021P-Tuning v2: Prompt Tuning Can Be Comparable to Fine-tuning Universally Across Scales and Taskscited by 686
  3. 2022RLPrompt: Optimizing Discrete Text Prompts with Reinforcement Learningcited by 361
  4. 2023Prompting Large Language Model for Machine Translation: A Case Studycited by 247
  5. 2022Delta Tuning: A Comprehensive Study of Parameter Efficient Methods for Pre-trained Language Modelscited by 179
  6. 2024Revisiting Zeroth-Order Optimization for Memory-Efficient LLM Fine-Tuning: A Benchmarkcited by 113
  7. 2024DoRA: Enhancing Parameter-Efficient Fine-Tuning with Dynamic Rank Distributioncited by 41
  8. 2022Decoupling Knowledge from Memorization: Retrieval-augmented Prompt Learningcited by 36
  9. 2023Hierarchical Prompting Assists Large Language Model on Web Navigationcited by 36
  10. 2023CodeCoT: Tackling Code Syntax Errors in CoT Reasoning for Code Generationcited by 29
  11. 2024TasTe: Teaching Large Language Models to Translate through Self-Reflectioncited by 27
  12. 2023Are Large Language Models Temporally Grounded?cited by 25
23 more
  1. 2022Prompt-Matched Semantic Segmentationcited by 22
  2. 2022TyDiP: A Dataset for Politeness Classification in Nine Typologically Diverse Languagescited by 21
  3. 2024A Novel Multi-Stage Prompting Approach for Language Agnostic MCQ Generation using GPTcited by 19
  4. 2024See Further for Parameter Efficient Fine-tuning by Standing on the Shoulders of Decompositioncited by 13
  5. 2024Exploring the Capabilities of Prompted Large Language Models in Educational and Assessment Applicationscited by 12
  6. 2022Contrastive Demonstration Tuning for Pre-trained Language Modelscited by 7
  7. 2022PromptDA: Label-guided Data Augmentation for Prompt-based Few-shot Learnerscited by 7
  8. 2024DeMPT: Decoding-enhanced Multi-phase Prompt Tuning for Making LLMs Be Better Context-aware Translatorscited by 6
  9. 2024Implicit Discourse Relation Classification For Nigerian Pidgincited by 6
  10. 2025CORRECT: Context- and Reference-Augmented Reasoning and Prompting for Fact-Checkingcited by 5
  11. 2023How Different Is Stereotypical Bias Across Languages?cited by 4
  12. 2024Attention Prompt Tuning: Parameter-efficient Adaptation of Pre-trained Models for Spatiotemporal Modelingcited by 4
  13. 2024Deconstructing In-Context Learning: Understanding Prompts via Corruptioncited by 4
  14. 2025FRAME: Boosting LLMs with A Four-Quadrant Multi-Stage Pretraining Strategycited by 4
  15. 2022Nebula-I: A General Framework for Collaboratively Training Deep Learning Models on Low-Bandwidth Cloud Clusterscited by 3
  16. 2026Do LLMs Really Memorize Personally Identifiable Information? Revisiting PII Leakage with a Cue-Controlled Memorization Frameworkcited by 3
  17. 2023LabelPrompt: Effective Prompt-based Learning for Relation Classificationcited by 1
  18. 2023Extrapolating Multilingual Understanding Models as Multilingual Generatorscited by 1
  19. 2023Domain Aligned Prefix Averaging for Domain Generalization in Abstractive Summarizationcited by 1
  20. 2023VBD-MT Chinese-Vietnamese Translation Systems for VLSP 2022cited by 1
  21. 2024Gradable ChatGPT Translation Evaluationcited by 1
  22. 2024UnifiedCrawl: Aggregated Common Crawl for Affordable Adaptation of LLMs on Low-Resource Languagescited by 1
  23. 2025Swin-TUNA : A Novel PEFT Approach for Accurate Food Image Segmentationcited by 1