mlsanityData to 7 Oct 2026

Searching for Effective Multilingual Fine-Tuning Methods: A Case Study in Summarization

arXiv 2212.05740 ↗First posted 12 Dec 2022cs.CLcited by 1

Not on the map. Why?

Abstract (arXiv)

Recently, a large number of tuning strategies have been proposed to adapt pre-trained language models to downstream tasks. In this paper, we perform an extensive empirical evaluation of various tuning strategies for multilingual learning, particularly in the context of text summarization. Specifically, we explore the relative advantages of three families of multilingual tuning strategies (a total of five models) and empirically evaluate them for summarization over 45 languages. Experimentally, we not only established a new state-of-the-art on the XL-Sum dataset but also derive a series of observations that hopefully can provide hints for future research on the design of multilingual tuning strategies.

Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Builds on 38 of its 51 references that are in the map. First posted on arXiv 12 Dec 2022.

Builds on

38 of 51 references.

References · 38

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  3. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  4. 2019BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehensioncited by 5,874
  5. 2019BERTScore: Evaluating Text Generation with BERTcited by 5,462
  6. 2019Parameter-Efficient Transfer Learning for NLPcited by 4,195
  7. 2019Unsupervised Cross-lingual Representation Learning at Scalecited by 3,907
  8. 2021Prefix-Tuning: Optimizing Continuous Prompts for Generationcited by 3,878
  9. 2021The Power of Scale for Parameter-Efficient Prompt Tuningcited by 3,769
  10. 2014Sequence to Sequence Learning with Neural Networkscited by 2,974
  11. 2021Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processingcited by 2,222
  12. 2020mT5: A massively multilingual pre-trained text-to-text transformercited by 1,813
26 more
  1. 2019PEGASUS: Pre-training with Extracted Gap-sentences for Abstractive Summarizationcited by 1,100
  2. 2019Cross-lingual Language Model Pretrainingcited by 1,088
  3. 2020Multilingual Denoising Pre-training for Neural Machine Translationcited by 1,028
  4. 2020COMET: A Neural Framework for MT Evaluationcited by 1,024
  5. 2021Towards a Unified View of Parameter-Efficient Transfer Learningcited by 863
  6. 2016Abstractive Text Summarization Using Sequence-to-Sequence RNNs and Beyondcited by 809
  7. 2021Compacter: Efficient Low-Rank Hypercomplex Adapter Layerscited by 458
  8. 2020MAD-X: An Adapter-Based Framework for Multi-Task Cross-Lingual Transfercited by 443
  9. 2020AdapterHub: A Framework for Adapting Transformerscited by 441
  10. 2015A Neural Attention Model for Abstractive Sentence Summarizationcited by 394
  11. 2020Parameter-Efficient Transfer Learning with Diff Pruningcited by 351
  12. 2021XL-Sum: Large-Scale Multilingual Abstractive Summarization for 44 Languagescited by 346
  13. 2021Pre-Trained Models: Past, Present and Futurecited by 319
  14. 2019Simple, Scalable Adaptation for Neural Machine Translationcited by 239
  15. 2015Semi-supervised Sequence Learningcited by 198
  16. 2019To Tune or Not to Tune? Adapting Pretrained Representations to Diverse Taskscited by 167
  17. 2021Noisy Channel Language Model Prompting for Few-Shot Text Classificationcited by 163
  18. 2021UniPELT: A Unified Framework for Parameter-Efficient Language Model Tuningcited by 162
  19. 2020GSum: A General Framework for Guided Neural Abstractive Summarizationcited by 129
  20. 2020MLSUM: The Multilingual Summarization Corpuscited by 121
  21. 2019What Would Elsa Do? Freezing Layers During Transformer Fine-Tuningcited by 87
  22. 2020Few-Shot Text Generation with Pattern-Exploiting Trainingcited by 87
  23. 2016One-shot Learning with Memory-Augmented Neural Networkscited by 60
  24. 2021Learning Language Specific Sub-network for Multilingual Machine Translationcited by 48
  25. 2020A Closer Look at Few-Shot Crosslingual Transfer: The Choice of Shots Matterscited by 45
  26. 2018Parameter Sharing Methods for Multilingual Self-Attentional Translation Modelscited by 33

Built on by

Cited by year

Citing papers per year: 2024: 1.

1 paper.

  1. 2024LLMs are Good Action Recognizerscited by 36