mlsanityData to 14 Oct 2026

Continual Pre-Training for Cross-Lingual LLM Adaptation: Enhancing Japanese Language Capabilities

arXiv 2404.17790 ↗First posted 27 Apr 2024cs.CL cs.AIcited by 111

Not on the map. Why?

Abstract (arXiv)

Cross-lingual continual pre-training of large language models (LLMs) initially trained on English corpus allows us to leverage the vast amount of English language resources and reduce the pre-training cost. In this study, we constructed Swallow, an LLM with enhanced Japanese capability, by extending the vocabulary of Llama 2 to include Japanese characters and conducting continual pre-training on a large Japanese web corpus. Experimental results confirmed that the performance on Japanese tasks drastically improved through continual pre-training, and the performance monotonically increased with the amount of training data up to 100B tokens. Consequently, Swallow achieved superior performance compared to other LLMs that were trained from scratch in English and Japanese. An analysis of the effects of continual pre-training revealed that it was particularly effective for Japanese question answering tasks. Furthermore, to elucidate effective methodologies for cross-lingual continual pre-training from English to Japanese, we investigated the impact of vocabulary expansion and the effectiveness of incorporating parallel corpora. The results showed that the efficiency gained through vocabulary expansion had no negative impact on performance, except for the summarization task, and that the combined use of parallel corpora enhanced translation ability.

Key facts. Cited by 111 papers in the map (data through 14 Oct 2026). Builds on 39 of its 55 references that are in the map. First posted on arXiv 27 Apr 2024.

Builds on

39 of 55 references.

References · 39

  1. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  2. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  3. 2016SQuAD: 100,000+ Questions for Machine Comprehension of Textcited by 4,148
  4. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,100
  5. 2017TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehensioncited by 2,955
  6. 2023FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioningcited by 2,598
  7. 2020The Pile: An 800GB Dataset of Diverse Text for Language Modelingcited by 2,411
  8. 2018Can a Suit of Armor Conduct Electricity? A New Dataset for Open Book Question Answeringcited by 2,164
  9. 2018SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processingcited by 1,989
  10. 2019BioBERT: a pre-trained biomedical language representation model for biomedical text miningcited by 1,947
  11. 2018CommonsenseQA: A Question Answering Challenge Targeting Commonsense Knowledgecited by 1,822
  12. 2018Know What You Don't Know: Unanswerable Questions for SQuADcited by 1,554
27 more
  1. 2019SciBERT: A Pretrained Language Model for Scientific Textcited by 1,414
  2. 2023The RefinedWeb Dataset for Falcon LLM: Outperforming Curated Corpora with Web Data, and Web Data Onlycited by 763
  3. 2020XTREME: A Massively Multilingual Multi-task Benchmark for Evaluating Cross-lingual Generalizationcited by 742
  4. 2020Language-agnostic BERT Sentence Embeddingcited by 627
  5. 2022Language Models are Multilingual Chain-of-Thought Reasonerscited by 530
  6. 2023Llemma: An Open Language Model For Mathematicscited by 388
  7. 2021XL-Sum: Large-Scale Multilingual Abstractive Summarization for 44 Languagescited by 347
  8. 2023Efficient and Effective Text Encoding for Chinese LLaMA and Alpacacited by 305
  9. 2022Reducing Activation Recomputation in Large Transformer Modelscited by 302
  10. 2023MEGA: Multilingual Evaluation of Generative AIcited by 293
  11. 2023Do All Languages Cost the Same? Tokenization in the Era of Commercial Language Modelscited by 192
  12. 2022Fine-tuned Language Models are Continual Learnerscited by 156
  13. 2023Continual Pre-Training of Large Language Models: How to (re)warm your model?cited by 137
  14. 2020Memory-Efficient Pipeline-Parallel DNN Trainingcited by 134
  15. 2021Lifelong Pretraining: Continually Adapting Language Models to Emerging Corporacited by 115
  16. 2024LLaMA Beyond English: An Empirical Study on Language Capability Transfercited by 95
  17. 2021XLM-E: Cross-lingual Language Model Pre-training via ELECTRAcited by 72
  18. 2021It's All in the Heads: Using Attention Heads as a Baseline for Cross-Lingual Transfer in Commonsense Reasoningcited by 69
  19. 2023Extrapolating Large Language Models to Non-English by Aligning Languagescited by 69
  20. 2023Searching for Needles in a Haystack: On the Role of Incidental Bilingualism in PaLM's Translation Capabilitycited by 62
  21. 2021Adapt-and-Distill: Developing Small, Fast and Effective Pretrained Language Models for Domainscited by 36
  22. 2023Empowering Cross-lingual Abilities of Instruction-tuned Large Language Models by Translation-following demonstrationscited by 35
  23. 2021Efficient Domain Adaptation of Language Models via Adaptive Tokenizationcited by 34
  24. 2023Language Versatilists vs. Specialists: An Empirical Revisiting on Multilingual Transfer Abilitycited by 27
  25. 2022JParaCrawl v3.0: A Large-scale English-Japanese Parallel Corpuscited by 17
  26. 2022Compositional Evaluation on Japanese Textual Entailment and Similaritycited by 14
  27. 2023Jamp: Controlled Japanese Temporal Inference Dataset for Evaluating Generalization Capacity of Language Modelscited by 4

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 32; 2025: 49; 2026: 30.

111 papers.

Sort:

By area:

  1. 2025Do Multilingual LLMs Think In English?cited by 78
  2. 2024LLaMAX: Scaling Linguistic Horizons of LLM by Enhancing Translation Capabilities Beyond 100 Languagescited by 48
  3. 2024Beyond English-Centric LLMs: What Language Do Multilingual Language Models Think in?cited by 41
  4. 2024Code-Switching Curriculum Learning for Multilingual Transfer in LLMscited by 27
  5. 2025Learn Globally, Speak Locally: Bridging the Gaps in Multilingual Reasoningcited by 24
  6. 2024How Can We Effectively Expand the Vocabulary of LLMs with 0.01GB of Target Language Text?cited by 22
  7. 2025R1-T1: Fully Incentivizing Translation Capability in LLMs via Reasoning Learningcited by 21
  8. 2025Rewriting Pre-Training Data Boosts LLM Performance in Math and Codecited by 21
  9. 2025The Emergence of Abstract Thought in Large Language Models Beyond Any Languagecited by 21
  10. 2025A Survey of Reasoning and Agentic Systems in Time Series with Large Language Modelscited by 21
  11. 2025Parity-Aware Byte-Pair Encoding: Improving Cross-lingual Fairness in Tokenizationcited by 20
  12. 2024Multilingual Large Language Models: A Systematic Surveycited by 18
38 more
  1. 2025CARE: Multilingual Human Preference Learning for Cultural Awarenesscited by 15
  2. 2024Exploring Reasoning Biases in Large Language Models Through Syllogism: Insights from the NeuBAROCO Datasetcited by 14
  3. 2025Massive Supervised Fine-tuning Experiments Reveal How Data, Layer, and Training Factors Shape LLM Alignment Qualitycited by 14
  4. 2025Revisiting Replay and Gradient Alignment for Continual Pre-Training of Large Language Modelscited by 14
  5. 2024Quality or Quantity? On Data Scale and Diversity in Adapting Large Language Models for Low-Resource Translationcited by 13
  6. 2025NileChat: Towards Linguistically Diverse and Culturally Aware LLMs for Local Communitiescited by 13
  7. 2025KokoroChat: A Japanese Psychological Counseling Dialogue Dataset Collected via Role-Playing by Trained Counselorscited by 13
  8. 2025One Tokenizer To Rule Them All: Emergent Language Plasticity via Multilingual Tokenizerscited by 12
  9. 2024Large Vocabulary Size Improves Large Language Modelscited by 11
  10. 2024Does Cross-Cultural Alignment Change the Commonsense Morality of Language Models?cited by 10
  11. 2025On the Consistency of Multilingual Context Utilization in Retrieval-Augmented Generationcited by 10
  12. 2025Rethinking Multilingual Continual Pretraining: Data Mixing for Adapting LLMs Across Languages and Resourcescited by 10
  13. 2024Evaluating the External and Parametric Knowledge Fusion of Large Language Modelscited by 9
  14. 2024Self-Distillation for Model Stacking Unlocks Cross-Lingual NLU in 200+ Languagescited by 9
  15. 2024JMedBench: A Benchmark for Evaluating Japanese Biomedical Large Language Modelscited by 8
  16. 2024Efficient Continual Pre-training of LLMs for Low-resource Languagescited by 8
  17. 2025Bring Your Own Knowledge: A Survey of Methods for LLM Knowledge Expansioncited by 8
  18. 2025Knowledge Fusion of Large Language Models Via Modular SkillPackscited by 8
  19. 2024The Power of Question Translation Training in Multilingual Reasoning: Broadened Scope and Deepened Insightscited by 7
  20. 2024JBBQ: Japanese Bias Benchmark for Analyzing Social Biases in Large Language Modelscited by 7
  21. 2024MergePrint: Merge-Resistant Fingerprints for Robust Black-box Ownership Verification of Large Language Modelscited by 7
  22. 2024Adapting Chat Language Models Using Only Target Unlabeled Language Datacited by 6
  23. 2026Beyond Static Models: An Evolving Framework for Continual Learning in Large Language Models across Training Stagescited by 6
  24. 2024InstructionCP: A fast approach to transfer Large Language Models into target languagecited by 5
  25. 2024Bilingual Adaptation of Monolingual Foundation Modelscited by 5
  26. 2025Emergent Abilities of Large Language Models under Continued Pretraining for Language Adaptationcited by 5
  27. 2024Pipeline Analysis for Developing Instruct LLMs in Low-Resource Languages: A Case Study on Basquecited by 4
  28. 2024Why We Build Local Large Language Models: An Observational Analysis from 35 Japanese and Multilingual LLMscited by 4
  29. 2025Massively Multilingual Adaptation of Large Language Models Using Bilingual Translation Datacited by 4
  30. 2025Addressing Tokenization Inconsistency in Steganography and Watermarking Based on Large Language Modelscited by 4
  31. 2025Mitigating Catastrophic Forgetting in Target Language Adaptation of LLMs via Source-Shielded Updatescited by 4
  32. 2024"Vorbecsti Romanecste?" A Recipe to Train Powerful Romanian LLMs with English Instructionscited by 3
  33. 2024Robust ASR Error Correction with Conservative Data Filteringcited by 3
  34. 2024Development and bilingual evaluation of Japanese medical large language model within reasonably low computational resourcescited by 3
  35. 2026OpenSeal: Good, Fast, and Cheap Construction of an Open-Source Southeast Asian LLM via Parallel Datacited by 3
  36. 2026Benchmarking Large Language Models for Grapheme-to-Phoneme Conversion: A Japanese Case Studycited by 3
  37. 2024Language Adaptation on a Tight Academic Compute Budget: Tokenizer Swapping Works and Pure bfloat16 Is Enoughcited by 2
  38. 2025Merging Language and Domain Specific Models: The Impact on Technical Vocabulary Acquisitioncited by 2

50 most cited of 111 All 111 (JSON)