mlsanityData to 7 Oct 2026

Generalization Measures for Zero-Shot Cross-Lingual Transfer

arXiv 2404.15928 ↗First posted 24 Apr 2024cs.CLcited by 0

Not on the map. Why?

Abstract (arXiv)

A model's capacity to generalize its knowledge to interpret unseen inputs with different characteristics is crucial to build robust and reliable machine learning systems. Language model evaluation tasks lack information metrics about model generalization and their applicability in a new setting is measured using task and language-specific downstream performance, which is often lacking in many languages and tasks. In this paper, we explore a set of efficient and reliable measures that could aid in computing more information related to the generalization capability of language models in cross-lingual zero-shot settings. In addition to traditional measures such as variance in parameters after training and distance from initialization, we also measure the effectiveness of sharpness in loss landscape in capturing the success in cross-lingual transfer and propose a novel and stable algorithm to reliably compute the sharpness of a model optimum that correlates to generalization.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 31 of its 44 references that are in the map. First posted on arXiv 24 Apr 2024.

Builds on

31 of 44 references.

References · 31

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  3. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  4. 2017A Broad-Coverage Challenge Corpus for Sentence Understanding through Inferencecited by 2,665
  5. 2020mT5: A massively multilingual pre-trained text-to-text transformercited by 1,813
  6. 2020Sharpness-Aware Minimization for Efficiently Improving Generalizationcited by 1,404
  7. 2016On Large-Batch Training for Deep Learning: Generalization Gap and Sharp Minimacited by 1,337
  8. 2021BitFit: Simple Parameter-efficient Fine-tuning for Transformer-based Masked Language-modelscited by 1,234
  9. 2018Averaging Weights Leads to Wider Optima and Better Generalizationcited by 1,194
  10. 2017Visualizing the Loss Landscape of Neural Netscited by 1,126
  11. 2019Cross-lingual Language Model Pretrainingcited by 1,088
  12. 2018XNLI: Evaluating Cross-lingual Sentence Representationscited by 913
19 more
  1. 2019Fantastic Generalization Measures and Where to Find Themcited by 507
  2. 2017Sharp Minima Can Generalize For Deep Netscited by 462
  3. 2021SWAD: Domain Generalization by Seeking Flat Minimacited by 431
  4. 2016Entropy-SGD: Biasing Gradient Descent Into Wide Valleyscited by 357
  5. 2019PAWS-X: A Cross-lingual Adversarial Dataset for Paraphrase Identificationcited by 262
  6. 2019SMART: Robust and Efficient Fine-Tuning for Pre-trained Natural Language Models through Principled Regularized Optimizationcited by 253
  7. 2021R-Drop: Regularized Dropout for Neural Networkscited by 202
  8. 2020Better Fine-Tuning by Reducing Representational Collapsecited by 156
  9. 2020Relative Flatness and Generalizationcited by 88
  10. 2022When Do Flat Minima Optimizers Work?cited by 78
  11. 2021Generalized Jensen-Shannon Divergence Loss for Learning with Noisy Labelscited by 74
  12. 2020Catastrophic Fisher Explosion: Early Phase Fisher Matrix Impacts Generalizationcited by 61
  13. 2019Generalization in Deep Networks: The Role of Distance from Initializationcited by 49
  14. 2021Consistency Regularization for Cross-Lingual Fine-Tuningcited by 41
  15. 2021Multi-view Subword Regularizationcited by 40
  16. 2021Noise Stability Regularization for Improving BERT Fine-tuningcited by 29
  17. 2021Improving Zero-Shot Cross-Lingual Transfer Learning via Robust Trainingcited by 21
  18. 2023Fantastic Generalization Measures are Nowhere to be Foundcited by 19
  19. 2021Consistency Training with Virtual Adversarial Discrete Perturbationcited by 9

Built on by

Not cited yet.