mlsanityData to 14 Oct 2026

Extrapolating Large Language Models to Non-English by Aligning Languages

arXiv 2308.04948 ↗First posted 9 Aug 2023cs.CLcited by 69

Not on the map. Why?

Abstract (arXiv)

Existing large language models show disparate capability across different languages, due to the imbalance in the training data. Their performances on English tasks are often stronger than on tasks of other languages. In this paper, we empower pre-trained LLMs on non-English languages by building semantic alignment across languages. We start from targeting individual languages by performing cross-lingual instruction-tuning (CoIT) on LLaMA, i.e. tuning it with translation task data and cross-lingual general task data to obtain cross-lingual models (x-LLaMAs), and formulate underlying scaling laws to investigate the advantages of using scalable translation data. Then we perform multilingual instruction-tuning (MuIT) with mixed resources to build multilingual m-LLaMA. We also illustrate how we leverage the scaling laws to optimize data allocation in a resource-constrained setting. Experiment results on cross-lingual benchmarks XQUAD and MLQA show that x-LLaMAs surpass the English instruction-tuned counterpart (Alpaca) by an average of 27.83% across six non-English languages. Evaluation results on translation dataset Flores-101 show that x-LLaMAs outperform previous LLaMA-based models by an average of 18.89%. Encouragingly, m-LLaMA achieves comparable performance to x-LLaMAs on individual languages and demonstrates the ability to follow multilingual instructions. Further analysis on response content and representation space reveals the alignment of the multilingual semantic space within the middle layers of m-LLaMA.

Key facts. Cited by 69 papers in the map (data through 14 Oct 2026). Builds on 34 of its 43 references that are in the map. First posted on arXiv 9 Aug 2023.

Builds on

34 of 43 references.

References · 34

  1. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  2. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  3. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,780
  4. 2022Self-Instruct: Aligning Language Models with Self-Generated Instructionscited by 2,724
  5. 2020The Pile: An 800GB Dataset of Diverse Text for Language Modelingcited by 2,411
  6. 2023G-Eval: NLG Evaluation using GPT-4 with Better Human Alignmentcited by 2,276
  7. 2020mT5: A massively multilingual pre-trained text-to-text transformercited by 1,818
  8. 2022No Language Left Behind: Scaling Human-Centered Machine Translationcited by 1,312
  9. 2020BLEURT: Learning Robust Metrics for Text Generationcited by 1,203
  10. 2023LIMA: Less Is More for Alignmentcited by 1,100
  11. 2023Large Language Models are not Fair Evaluatorscited by 1,085
  12. 2015Aligning Books and Movies: Towards Story-like Visual Explanations by Watching Movies and Reading Bookscited by 1,043
22 more
  1. 2020COMET: A Neural Framework for MT Evaluationcited by 1,033
  2. 2023A Multitask, Multilingual, Multimodal Evaluation of ChatGPT on Reasoning, Hallucination, and Interactivitycited by 894
  3. 2022Super-NaturalInstructions: Generalization via Declarative Instructions on 1600+ NLP Taskscited by 868
  4. 2023The Flan Collection: Designing Data and Methods for Effective Instruction Tuningcited by 739
  5. 2023C-Eval: A Multi-Level Multi-Discipline Chinese Evaluation Suite for Foundation Modelscited by 708
  6. 2023PyTorch FSDP: Experiences on Scaling Fully Sharded Data Parallelcited by 660
  7. 2019On the Cross-lingual Transferability of Monolingual Representationscited by 578
  8. 2020Beyond English-Centric Multilingual Machine Translationcited by 561
  9. 2021The FLORES-101 Evaluation Benchmark for Low-Resource and Multilingual Machine Translationcited by 560
  10. 2023How Far Can Camels Go? Exploring the State of Instruction Tuning on Open Resourcescited by 408
  11. 2019MLQA: Evaluating Cross-lingual Extractive Question Answeringcited by 340
  12. 2023Efficient and Effective Text Encoding for Chinese LLaMA and Alpacacited by 305
  13. 2019WikiMatrix: Mining 135M Parallel Sentences in 1620 Language Pairs from Wikipediacited by 223
  14. 2023Not All Languages Are Created Equal in LLMs: Improving Multilingual Capability by Cross-Lingual-Thought Promptingcited by 165
  15. 2023Multilingual Machine Translation with Large Language Models: Empirical Results and Analysiscited by 160
  16. 2021Contrastive Learning for Many-to-many Multilingual Neural Machine Translationcited by 108
  17. 2022Language Contamination Helps Explain the Cross-lingual Capabilities of English Pretrained Modelscited by 92
  18. 2023KoLA: Carefully Benchmarking World Knowledge of Large Language Modelscited by 85
  19. 2023Eliciting the Translation Ability of Large Language Models via Multilingual Finetuning with Translation Instructionscited by 56
  20. 2023BayLing: Bridging Cross-lingual Alignment and Instruction Following through Interactive Translation for Large Language Modelscited by 42
  21. 2020From English To Foreign Languages: Transferring Pre-trained Language Modelscited by 37
  22. 2023ParroT: Translating during Chat using Large Language Models tuned with Human Translation and Feedbackcited by 34

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 7; 2024: 43; 2025: 17; 2026: 2.

69 papers.

Sort:

By area:

  1. 2024Contrastive Preference Optimization: Pushing the Boundaries of LLM Performance in Machine Translationcited by 394
  2. 2024Do Llamas Work in English? On the Latent Language of Multilingual Transformerscited by 281
  3. 2024Continual Pre-Training for Cross-Lingual LLM Adaptation: Enhancing Japanese Language Capabilitiescited by 111
  4. 2024Large Language Models Meet NLP: A Surveycited by 103
  5. 2023SeaEval for Multilingual Foundation Models: From Cross-Lingual Alignment to Cultural Reasoningcited by 94
  6. 2023A Paradigm Shift in Machine Translation: Boosting Translation Performance of Large Language Modelscited by 89
  7. 2025Do Multilingual LLMs Think In English?cited by 78
  8. 2023Monolingual or Multilingual Instruction Tuning: Which Makes a Better Alpacacited by 60
  9. 2024Multilingual Large Language Model: A Survey of Resources, Taxonomy and Frontierscited by 51
  10. 2024LLMs Beyond English: Scaling the Multilingual Capability of LLMs with Cross-Lingual Feedbackcited by 51
  11. 2024Question Translation Training for Better Multilingual Reasoningcited by 42
  12. 2023PLUG: Leveraging Pivot Language in Cross-Lingual Instruction Tuningcited by 39
38 more
  1. 2023Empowering Cross-lingual Abilities of Instruction-tuned Large Language Models by Translation-following demonstrationscited by 35
  2. 2024AdaMergeX: Cross-Lingual Transfer with Large Language Models via Adaptive Adapter Mergingcited by 34
  3. 2024MaLA-500: Massive Language Adaptation of Large Language Modelscited by 28
  4. 2024Teaching Llama a New Language Through Cross-Lingual Knowledge Transfercited by 23
  5. 2024LangBridge: Multilingual Reasoning Without Multilingual Supervisioncited by 22
  6. 2024The Fine-Tuning Paradox: Boosting Translation Quality Without Sacrificing LLM Abilitiescited by 22
  7. 2024Multilingual Large Language Models: A Systematic Surveycited by 18
  8. 2024Salute the Classic: Revisiting Challenges of Machine Translation in the Age of Large Language Modelscited by 17
  9. 2025AdaMCoT: Rethinking Cross-Lingual Factual Reasoning through Adaptive Multilingual Chain-of-Thoughtcited by 14
  10. 2024mEdIT: Multilingual Text Editing via Instruction Tuningcited by 13
  11. 2024Adapting LLMs to Hebrew: Unveiling DictaLM 2.0 with Enhanced Vocabulary and Instruction Capabilitiescited by 13
  12. 2024Quality or Quantity? On Data Scale and Diversity in Adapting Large Language Models for Low-Resource Translationcited by 13
  13. 2023How Vocabulary Sharing Facilitates Multilingualism in LLaMA?cited by 12
  14. 2024CrossIn: An Efficient Instruction Tuning Approach for Cross-Lingual Knowledge Alignmentcited by 11
  15. 2025Enhancing Non-English Capabilities of English-Centric Large Language Models through Deep Supervision Fine-Tuningcited by 11
  16. 2025Improving Multilingual Language Models by Aligning Representations through Steeringcited by 11
  17. 2025Aligning LLMs for Multilingual Consistency in Enterprise Applicationscited by 11
  18. 2024Aurora-M: Open Source Continual Pre-training for Multilingual Language and Codecited by 10
  19. 2024Crosslingual Capabilities and Knowledge Barriers in Multilingual Large Language Modelscited by 9
  20. 2024Facilitating large language model Russian adaptation with Learned Embedding Propagationcited by 8
  21. 2025Mechanistic Understanding and Mitigation of Language Confusion in English-Centric Large Language Modelscited by 8
  22. 2025Tracing Multilingual Representations in LLMs with Cross-Layer Transcoderscited by 8
  23. 2024The Power of Question Translation Training in Multilingual Reasoning: Broadened Scope and Deepened Insightscited by 7
  24. 2024Lens: Rethinking Multilingual Enhancement for Large Language Modelscited by 6
  25. 2025AlignX: Advancing Multilingual Large Language Models with Multilingual Representation Alignmentcited by 6
  26. 2024CourseGPT-zh: an Educational Large Language Model Based on Knowledge Distillation Incorporating Prompt Optimizationcited by 5
  27. 2024Learning vs Retrieval: The Role of In-Context Examples in Regression with Large Language Modelscited by 5
  28. 2024Guardians of Discourse: Evaluating LLMs on Multilingual Offensive Language Detectioncited by 5
  29. 2025Document Intelligence in the Era of Large Language Models: A Surveycited by 5
  30. 2023Danish Foundation Modelscited by 4
  31. 2024Selected Languages are All You Need for Cross-lingual Truthfulness Transfercited by 4
  32. 2024Aligning Translation-Specific Understanding to General Understanding in Large Language Modelscited by 3
  33. 2024An Empirical Study on Large Language Models in Accuracy and Robustness under Chinese Industrial Scenarioscited by 3
  34. 2024KS-Lottery: Finding Certified Lottery Tickets for Multilingual Language Modelscited by 3
  35. 2024Thinker-DDM: Modeling Deliberation for Machine Translation with a Drift-Diffusion Processcited by 3
  36. 2024Enhancing Travel Choice Modeling with Large Language Models: A Prompt-Learning Approachcited by 3
  37. 2024LLMs for Extremely Low-Resource Finno-Ugric Languagescited by 3
  38. 2024KnowCoder-X: Boosting Multilingual Information Extraction via Codecited by 3

50 most cited of 69 All 69 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024Getting More from Less: Large Language Models are Good Spontaneous Multilingual Learners
  2. 2025Middle-Layer Representation Alignment for Cross-Lingual Transfer in Fine-Tuned LLMs
  3. 2024Multilingual Pretraining and Instruction Tuning Improve Cross-Lingual Knowledge Alignment, But Only Shallowly
  4. 2024BayLing 2: A Multilingual Large Language Model with Efficient Language Alignment
  5. 2025Aligning Large Language Models to Low-Resource Languages through LLM-Based Selective Translation: A Systematic Study
  6. 2025Towards Alignment-Centric Paradigm: A Survey of Instruction Tuning in Large Language Models
  7. 2024Pruning Multilingual Large Language Models for Multilingual Inference
  8. 2023InstructAlign: High-and-Low Resource Language Alignment via Continual Crosslingual Instruction Tuning