mlsanityData to 7 Oct 2026

LEIA: Facilitating Cross-lingual Knowledge Transfer in Language Models with Entity-based Data Augmentation

arXiv 2402.11485 ↗First posted 18 Feb 2024cs.CL cs.AI cs.LGcited by 1

Not on the map. Why?

Abstract (arXiv)

Adapting English-based large language models (LLMs) to other languages has become increasingly popular due to the efficiency and potential of cross-lingual transfer. However, existing language adaptation methods often overlook the benefits of cross-lingual supervision. In this study, we introduce LEIA, a language adaptation tuning method that utilizes Wikipedia entity names aligned across languages. This method involves augmenting the target language corpus with English entity names and training the model using left-to-right language modeling. We assess LEIA on diverse question answering datasets using 7B-parameter LLMs, demonstrating significant performance gains across various non-English languages. The source code is available at https://github.com/studio-ousia/leia.

Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Builds on 24 of its 38 references that are in the map. First posted on arXiv 18 Feb 2024.

Builds on

24 of 38 references.

References · 24

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2023FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioningcited by 2,551
  3. 2018CommonsenseQA: A Question Answering Challenge Targeting Commonsense Knowledgecited by 1,793
  4. 2017Decoupled Weight Decay Regularizationcited by 1,722
  5. 2020Don't Stop Pretraining: Adapt Language Models to Domains and Taskscited by 1,548
  6. 2019ZeRO: Memory Optimizations Toward Training Trillion Parameter Modelscited by 1,278
  7. 2019Cross-lingual Language Model Pretrainingcited by 1,088
  8. 2020The State and Fate of Linguistic Diversity and Inclusion in the NLP Worldcited by 789
  9. 2023Efficient and Effective Text Encoding for Chinese LLaMA and Alpacacited by 302
  10. 2023MEGA: Multilingual Evaluation of Generative AIcited by 291
  11. 2019Emerging Cross-lingual Structure in Pretrained Language Modelscited by 180
  12. 2023Not All Languages Are Created Equal in LLMs: Improving Multilingual Capability by Cross-Lingual-Thought Promptingcited by 163
12 more
  1. 2023SeaLLMs -- Large Language Models for Southeast Asiacited by 104
  2. 2024LLaMA Beyond English: An Empirical Study on Language Capability Transfercited by 94
  3. 2022BLOOM+1: Adding Language Support to BLOOM for Zero-Shot Promptingcited by 90
  4. 2020Extending Multilingual BERT to Low-Resource Languagescited by 88
  5. 2023Do Multilingual Language Models Think Better in English?cited by 87
  6. 2018Rapid Adaptation of Neural Machine Translation to New Languagescited by 82
  7. 2021Common Sense Beyond English: Evaluating and Improving Multilingual Language Models for Commonsense Reasoningcited by 64
  8. 2022Expanding Pretrained Models to Thousands More Languages via Lexicon-based Adaptationcited by 44
  9. 2021nmT5 -- Is parallel data still relevant for pre-training massively multilingual language models?cited by 21
  10. 2021PARADISE: Exploiting Parallel Data for Multilingual Sequence-to-Sequence Pretrainingcited by 21
  11. 2022Cedille: A large autoregressive French language modelcited by 20
  12. 2021XLM-K: Improving Cross-Lingual Language Model Pre-training with Multilingual Knowledgecited by 18

Built on by

Cited by year

Citing papers per year: 2025: 1.

1 paper.

  1. 2025Tracing Multilingual Knowledge Acquisition Dynamics in Domain Adaptation: A Case Study of English-Japanese Biomedical Adaptationcited by 0