Measuring Catastrophic Forgetting in Cross-Lingual Transfer Paradigms: Exploring Tuning Strategies

The cross-lingual transfer is a promising technique to solve tasks in less-resourced languages. In this empirical study, we compare two fine-tuning approaches combined with zero-shot and full-shot learning approaches for large language models in a cross-lingual setting. As fine-tuning strategies, we compare parameter-efficient adapter methods with fine-tuning of all parameters. As cross-lingual transfer strategies, we compare the intermediate-training (textitIT) that uses each language sequentially and cross-lingual validation (textitCLV) that uses a target language already in the validation phase of fine-tuning. We assess the success of transfer and the extent of catastrophic forgetting in a source language due to cross-lingual transfer, i.e., how much previously acquired knowledge is lost when we learn new information in a different language. The results on two different classification problems, hate speech detection and product reviews, each containing datasets in several languages, show that the textitIT cross-lingual strategy outperforms textitCLV for the target language. Our findings indicate that, in the majority of cases, the textitCLV strategy demonstrates superior retention of knowledge in the base language (English) compared to the textitIT strategy, when evaluating catastrophic forgetting in multiple cross-lingual transfers.