mlsanityData to 7 Oct 2026

Teaching Llama a New Language Through Cross-Lingual Knowledge Transfer

arXiv 2404.04042 ↗First posted 5 Apr 2024cs.CLcited by 23

Not on the map. Why?

Abstract (arXiv)

This paper explores cost-efficient methods to adapt pretrained Large Language Models (LLMs) to new lower-resource languages, with a specific focus on Estonian. Leveraging the Llama 2 model, we investigate the impact of combining cross-lingual instruction-tuning with additional monolingual pretraining. Our results demonstrate that even a relatively small amount of additional monolingual pretraining followed by cross-lingual instruction-tuning significantly enhances results on Estonian. Furthermore, we showcase cross-lingual knowledge transfer from high-quality English instructions to Estonian, resulting in improvements in commonsense reasoning and multi-turn conversation capabilities. Our best model, named textscLlammas, represents the first open-source instruction-following LLM for Estonian. Additionally, we publish Alpaca-est, the first general task instruction dataset for Estonia. These contributions mark the initial progress in the direction of developing open-source LLMs for Estonian.

Key facts. Cited by 23 papers in the map (data through 7 Oct 2026). Builds on 29 of its 56 references that are in the map. First posted on arXiv 5 Apr 2024.

Builds on

29 of 56 references.

References · 29

  1. 2022Training language models to follow instructions with human feedbackcited by 15,712
  2. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  3. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  4. 2020Measuring Massive Multitask Language Understandingcited by 7,768
  5. 2016SQuAD: 100,000+ Questions for Machine Comprehension of Textcited by 4,115
  6. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,018
  7. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,747
  8. 2021TruthfulQA: Measuring How Models Mimic Human Falsehoodscited by 2,919
  9. 2017TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehensioncited by 2,895
  10. 2022Self-Instruct: Aligning Language Models with Self-Generated Instructionscited by 2,674
  11. 2022BLOOM: A 176B-Parameter Open-Access Multilingual Language Modelcited by 1,968
  12. 2018A Call for Clarity in Reporting BLEU Scorescited by 1,790
17 more
  1. 2021Multitask Prompted Training Enables Zero-Shot Task Generalizationcited by 1,462
  2. 2023LIMA: Less Is More for Alignmentcited by 1,080
  3. 2020COMET: A Neural Framework for MT Evaluationcited by 1,024
  4. 2023OpenAssistant Conversations -- Democratizing Large Language Model Alignmentcited by 712
  5. 2021Cross-Task Generalization via Natural Language Crowdsourcing Instructionscited by 696
  6. 2019On the Cross-lingual Transferability of Monolingual Representationscited by 575
  7. 2023How Far Can Camels Go? Exploring the State of Instruction Tuning on Open Resourcescited by 405
  8. 2023Efficient and Effective Text Encoding for Chinese LLaMA and Alpacacited by 302
  9. 2023The False Promise of Imitating Proprietary LLMscited by 254
  10. 2023Camels in a Changing Climate: Enhancing LM Adaptation with Tulu 2cited by 227
  11. 2019WikiMatrix: Mining 135M Parallel Sentences in 1620 Language Pairs from Wikipediacited by 221
  12. 2022BLOOM+1: Adding Language Support to BLOOM for Zero-Shot Promptingcited by 90
  13. 2023Extrapolating Large Language Models to Non-English by Aligning Languagescited by 69
  14. 2023PolyLM: An Open Source Polyglot Large Language Modelcited by 64
  15. 2018Approaching Neural Grammatical Error Correction as a Low-Resource Machine Translation Taskcited by 56
  16. 2023PLUG: Leveraging Pivot Language in Cross-Lingual Instruction Tuningcited by 39
  17. 2023Empowering Cross-lingual Abilities of Instruction-tuned Large Language Models by Translation-following demonstrationscited by 34

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 11; 2025: 6; 2026: 6.

23 papers.

Sort:

By area:

  1. 2024Bridging the Language Gaps in Large Language Models with Inference-Time Cross-Lingual Interventioncited by 24
  2. 2025The Emergence of Abstract Thought in Large Language Models Beyond Any Languagecited by 20
  3. 2024MoE-LPR: Multilingual Extension of Large Language Models through Mixture-of-Experts with Language Priors Routingcited by 16
  4. 2024Facilitating large language model Russian adaptation with Learned Embedding Propagationcited by 8
  5. 2024RoQLlama: A Lightweight Romanian Adapted Language Modelcited by 7
  6. 2024LlamaTurk: Adapting Open-Source Generative Large Language Models for Low-Resource Languagecited by 6
  7. 2024Lens: Rethinking Multilingual Enhancement for Large Language Modelscited by 5
  8. 2024Pipeline Analysis for Developing Instruct LLMs in Low-Resource Languages: A Case Study on Basquecited by 4
  9. 2025Understanding LLMs' Cross-Lingual Context Retrieval: How Good It Is And Where It Comes Fromcited by 4
  10. 2024LLMs for Extremely Low-Resource Finno-Ugric Languagescited by 3
  11. 2024GemmAr: Enhancing LLMs Through Arabic Instruction-Tuningcited by 2
  12. 2024Language Adaptation on a Tight Academic Compute Budget: Tokenizer Swapping Works and Pure bfloat16 Is Enoughcited by 2
11 more
  1. 2025Uncovering inequalities in new knowledge learning by large language models across different languagescited by 1
  2. 2025Improving Multilingual Capabilities with Cultural and Local Knowledge in Large Language Models While Enhancing Native Performancecited by 1
  3. 2026EstLLM: Enhancing Estonian Capabilities in Multilingual LLMs via Continued Pretraining and Post-Trainingcited by 1
  4. 2026PolyFact: Comparing Consistency-Driven Post-training Methods for Cross-Lingual Factual Recallcited by 1
  5. 2024NusaMT-7B: Machine Translation for Low-Resource Indonesian Languages with Large Language Modelscited by 0
  6. 2025Prompt, Translate, Fine-Tune, Re-Initialize, or Instruction-Tune? Adapting LLMs for In-Context Learning in Low-Resource Languagescited by 0
  7. 2025Persian-Phi: Efficient Cross-Lingual Adaptation of Compact LLMs via Curriculum Learningcited by 0
  8. 2026Less Languages, Less Tokens: An Efficient Unified Logic Cross-lingual Chain-of-Thought Reasoning Frameworkcited by 0
  9. 2026Repetition over Diversity: High-Signal Data Filtering for Sample-Efficient German Language Modelingcited by 0
  10. 2026MameLoshnLM: Yiddish Language Model and Evaluation Benchmarkcited by 0
  11. 2026From Zero to Hero: An Open LLM Ecosystem for Armeniancited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2023TaCo: Enhancing Cross-Lingual Transfer for Low-Resource Languages in LLMs through Translation-Assisted Chain-of-Thought Processes
  2. 2024LLMs Beyond English: Scaling the Multilingual Capability of LLMs with Cross-Lingual Feedback
  3. 2024X-Instruction: Aligning Language Model in Low-resource Languages with Self-curated Cross-lingual Instructions
  4. 2024Multilingual Pretraining and Instruction Tuning Improve Cross-Lingual Knowledge Alignment, But Only Shallowly
  5. 2025Got Compute, but No Data: Lessons From Post-training a Finnish LLM
  6. 2023BayLing: Bridging Cross-lingual Alignment and Instruction Following through Interactive Translation for Large Language Models
  7. 2024SambaLingo: Teaching Large Language Models New Languages
  8. 2024CrossIn: An Efficient Instruction Tuning Approach for Cross-Lingual Knowledge Alignment