mlsanityData to 14 Oct 2026

GradSim: Gradient-Based Language Grouping for Effective Multilingual Training

arXiv 2310.15269 ↗First posted 23 Oct 2023cs.LG cs.CLcited by 6

Not on the map. Why?

Abstract (arXiv)

Most languages of the world pose low-resource challenges to natural language processing models. With multilingual training, knowledge can be shared among languages. However, not all languages positively influence each other and it is an open research question how to select the most suitable set of languages for multilingual training and avoid negative interference among languages whose characteristics or data distributions are not compatible. In this paper, we propose GradSim, a language grouping method based on gradient similarity. Our experiments on three diverse multilingual benchmark datasets show that it leads to the largest performance gains compared to other similarity measures and it is better correlated with cross-lingual model performance. As a result, we set the new state of the art on AfriSenti, a benchmark dataset for sentiment analysis on low-resource African languages. In our extensive analysis, we further reveal that besides linguistic features, the topics of the datasets play an important role for language grouping and that lower layers of transformer models encode language-specific features while higher layers capture task-specific information.

Key facts. Cited by 6 papers in the map (data through 14 Oct 2026). Builds on 31 of its 42 references that are in the map. First posted on arXiv 23 Oct 2023.

Builds on

31 of 42 references.

References · 31

  1. 2020Don't Stop Pretraining: Adapt Language Models to Domains and Taskscited by 1,565
  2. 2020Gradient Surgery for Multi-Task Learningcited by 1,256
  3. 2019BERT Rediscovers the Classical NLP Pipelinecited by 1,059
  4. 2017GradNorm: Gradient Normalization for Adaptive Loss Balancing in Deep Multitask Networkscited by 834
  5. 2018Multi-Task Learning as Multi-Objective Optimizationcited by 821
  6. 2017An Overview of Multi-Task Learning in Deep Neural Networkscited by 803
  7. 2020XTREME: A Massively Multilingual Multi-task Benchmark for Evaluating Cross-lingual Generalizationcited by 742
  8. 2019How multilingual is Multilingual BERT?cited by 645
  9. 2018Taskonomy: Disentangling Task Transfer Learningcited by 634
  10. 2019Which Tasks Should Be Learned Together in Multi-task Learning?cited by 375
  11. 2019Revealing the Dark Secrets of BERTcited by 301
  12. 2021Efficiently Identifying Task Groupings for Multi-Task Learningcited by 222
19 more
  1. 2020A Survey on Recent Approaches for Natural Language Processing in Low-Resource Scenarioscited by 171
  2. 2020Gradient Vaccine: Investigating and Improving Multi-task Optimization in Massively Multilingual Modelscited by 169
  3. 2022Adapting Pre-trained Language Models to African Languages via Multilingual Adaptive Fine-Tuningcited by 149
  4. 2019Choosing Transfer Languages for Cross-Lingual Learningcited by 144
  5. 2022The Geometry of Multilingual Language Model Representationscited by 95
  6. 2023AfriSenti: A Twitter Sentiment Analysis Benchmark for African Languagescited by 68
  7. 2019Multilingual Neural Machine Translation with Language Clusteringcited by 47
  8. 2023SemEval-2023 Task 12: Sentiment Analysis for African Languages (AfriSenti-SemEval)cited by 41
  9. 2022A Balanced Data Approach for Evaluating Cross-Lingual Transfer: Mapping the Linguistic Blood Bankcited by 29
  10. 2017Robust Multilingual Part-of-Speech Tagging via Adversarial Trainingcited by 21
  11. 2020Bridging Linguistic Typology and Multilingual Machine Translation with Multi-View Language Representationscited by 21
  12. 2023Transfer to a Low-Resource Language via Close Relatives: The Case Study on Faroesecited by 15
  13. 2022Language-Family Adapters for Low-Resource Multilingual Neural Machine Translationcited by 14
  14. 2020Adversarial Alignment of Multilingual Models for Extracting Temporal Expressions from Textcited by 11
  15. 2021To Share or not to Share: Predicting Sets of Sources for Model Transfer Learningcited by 11
  16. 2023A Crosslingual Investigation of Conceptualization in 1335 Languagescited by 11
  17. 2023NLNDE at SemEval-2023 Task 12: Adaptive Pretraining and Source Language Selection for Low-Resource Multilingual Sentiment Analysiscited by 10
  18. 2022Por Qu'e N~ao Utiliser Alla Spraak? Mixed Training with Gradient Optimization in Few-Shot Cross-Lingual Transfercited by 7
  19. 2020FAME: Feature-Based Adversarial Meta-Embeddings for Robust Input Representationscited by 4

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 3; 2025: 0; 2026: 3.

6 papers.

  1. 2024Rehearsal-Free Modular and Compositional Continual Learning for Language Modelscited by 27
  2. 2024Learn it or Leave it: Module Composition and Pruning for Continual Learningcited by 3
  3. 2024LangSAMP: Language-Script Aware Multilingual Pretrainingcited by 2
  4. 2026Structural priors for data-efficient language learningcited by 1
  5. 2026Why Better Cross-Lingual Alignment Fails for Better Cross-Lingual Transfer: Case of Encoderscited by 0
  6. 2026Understanding LLM Behavior in Multi-Target Cross-Lingual Summarizationcited by 0