mlsanityData to 7 Oct 2026

InstructAlign: High-and-Low Resource Language Alignment via Continual Crosslingual Instruction Tuning

arXiv 2305.13627 ↗First posted 23 May 2023cs.CL cs.AIcited by 21

Not on the map. Why?

Abstract (arXiv)

Large language models (LLMs) that are tuned with instructions have demonstrated remarkable capabilities in various tasks and languages. However, their ability to generalize to underrepresented languages is limited due to the scarcity of available data. Additionally, directly adapting new languages to instruction-tuned LLMs can result in catastrophic forgetting, which leads to the loss of multitasking ability. To address this issue, we propose InstructAlign which uses continual crosslingual instruction tuning to enable LLMs to align new unseen languages with previously learned high-resource languages. Our results demonstrate the effectiveness of InstructAlign in enabling the model to understand low-resource languages with limited parallel data while preventing catastrophic forgetting. Our work contributes to the advancement of language adaptation methods, particularly for adapting instruction-tuned LLMs to underrepresented languages. Our code is released on https://github.com/HLTCHKUST/InstructAlign

Key facts. Cited by 21 papers in the map (data through 7 Oct 2026). Builds on 43 of its 76 references that are in the map. First posted on arXiv 23 May 2023.

Builds on

43 of 76 references.

References · 43

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2022Training language models to follow instructions with human feedbackcited by 15,712
  4. 2020Scaling Laws for Neural Language Modelscited by 6,411
  5. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
  6. 2016Overcoming catastrophic forgetting in neural networkscited by 4,718
  7. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,747
  8. 2017Gradient Episodic Memory for Continual Learningcited by 1,871
  9. 2020mT5: A massively multilingual pre-trained text-to-text transformercited by 1,813
  10. 2022No Language Left Behind: Scaling Human-Centered Machine Translationcited by 1,290
  11. 2017Mixed Precision Trainingcited by 1,135
  12. 2017Memory Aware Synapses: Learning what (not) to forgetcited by 1,094
31 more
  1. 2019Cross-lingual Language Model Pretrainingcited by 1,088
  2. 2018Experience Replay for Continual Learningcited by 1,074
  3. 2018Efficient Lifelong Learning with A-GEMcited by 1,044
  4. 2022Few-Shot Parameter-Efficient Fine-Tuning is Better and Cheaper than In-Context Learningcited by 886
  5. 2017PackNet: Adding Multiple Tasks to a Single Network by Iterative Pruningcited by 882
  6. 2022GLM-130B: An Open Bilingual Pre-trained Modelcited by 837
  7. 2018Overcoming catastrophic forgetting with hard attention to the taskcited by 672
  8. 2022Using DeepSpeed and Megatron to Train Megatron-Turing NLG 530B, A Large-Scale Generative Language Modelcited by 557
  9. 2021The FLORES-101 Evaluation Benchmark for Low-Resource and Multilingual Machine Translationcited by 556
  10. 2017Word Translation Without Parallel Datacited by 480
  11. 2020MAD-X: An Adapter-Based Framework for Multi-Task Cross-Lingual Transfercited by 443
  12. 2016Expert Gate: Lifelong Learning with a Network of Expertscited by 369
  13. 2017Unsupervised Machine Translation Using Monolingual Corpora Onlycited by 272
  14. 2021MasakhaNER: Named Entity Recognition for African Languagescited by 155
  15. 2022Adapting Pre-trained Language Models to African Languages via Multilingual Adaptive Fine-Tuningcited by 149
  16. 2020When Being Unseen from mBERT is just the Beginning: Handling New Languages With Multilingual Language Modelscited by 131
  17. 2018Rotate your Networks: Better Weight Consolidation and Less Catastrophic Forgettingcited by 127
  18. 2020Multilingual Alignment of Contextual Word Representationscited by 113
  19. 2020IndoNLU: Benchmark and Resources for Evaluating Indonesian Natural Language Understandingcited by 108
  20. 2022BLOOM+1: Adding Language Support to BLOOM for Zero-Shot Promptingcited by 90
  21. 2022One Country, 700+ Languages: NLP Challenges for Underrepresented Languages and Dialects in Indonesiacited by 65
  22. 2020IndoLEM and IndoBERT: A Benchmark Dataset and Pre-trained Language Model for Indonesian NLPcited by 57
  23. 2021IndoNLG: Benchmark and Resources for Evaluating Indonesian Natural Language Generationcited by 57
  24. 2021How to Adapt Your Pretrained Multilingual Model to 1600 Languagescited by 54
  25. 2021AmericasNLI: Evaluating Zero-shot Natural Language Understanding of Pretrained Multilingual Models in Truly Low-resource Languagescited by 50
  26. 2023Prompting Multilingual Large Language Models to Generate Code-Mixed Texts: The Case of South East Asian Languagescited by 50
  27. 2022IndicNLG Benchmark: Multilingual Datasets for Diverse NLG Tasks in Indic Languagescited by 42
  28. 2017Learning Language Representations for Typology Predictioncited by 40
  29. 2022GEMv2: Multilingual NLG Benchmarking in a Single Line of Codecited by 36
  30. 2021IndoNLI: A Natural Language Inference Dataset for Indonesiancited by 24
  31. 2020Improving Bi-LSTM Performance for Indonesian Sentiment Analysis Using Paragraph Vectorcited by 18

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 11; 2025: 7; 2026: 3.

21 papers.

Sort:

By area:

  1. 2024Recent Advances of Foundation Language Models-based Continual Learning: A Surveycited by 46
  2. 2024Language Model Alignment in Multilingual Trolley Problemscited by 46
  3. 2024Teaching Large Language Models an Unseen Language on the Flycited by 29
  4. 2024Multilingual Pretraining and Instruction Tuning Improve Cross-Lingual Knowledge Alignment, But Only Shallowlycited by 27
  5. 2024An Empirical Comparison of Vocabulary Expansion and Initialization Approaches for Language Modelscited by 20
  6. 2024Unlocking the Potential of Model Merging for Low-Resource Languagescited by 19
  7. 2024Cendol: Open Instruction-tuned Generative Large Language Models for Indonesian Languagescited by 18
  8. 2024Multilingual Large Language Models: A Systematic Surveycited by 18
  9. 2026Align Once, Benefit Multilingually: Enforcing Multilingual Consistency for LLM Safety Alignmentcited by 9
  10. 2025Crowdsource, Crawl, or Generate? Creating SEA-VL, a Multicultural Vision-Language Dataset for Southeast Asiacited by 8
  11. 2024Thank You, Stingray: Multilingual Large Language Models Can Not (Yet) Disambiguate Cross-Lingual Word Sensecited by 4
  12. 2025Instruction Tuning on Public Government and Cultural Data for Low-Resource Language: a Case Study in Kazakhcited by 4
9 more
  1. 2025Understanding LLMs' Cross-Lingual Context Retrieval: How Good It Is And Where It Comes Fromcited by 4
  2. 2025Aligning Large Language Models to Low-Resource Languages through LLM-Based Selective Translation: A Systematic Studycited by 4
  3. 2025High-Dimensional Interlingual Representations of Large Language Modelscited by 2
  4. 2024WangchanLion and WangchanX MRC Evalcited by 0
  5. 2024Exploring Continual Fine-Tuning for Enhancing Language Ability in Large Language Modelcited by 0
  6. 2025Entropy2Vec: Crosslingual Language Modeling Entropy as End-to-End Learnable Language Representationscited by 0
  7. 2025CUTE: A Multilingual Dataset for Enhancing Cross-Lingual Knowledge Transfer in Low-Resource Languagescited by 0
  8. 2026Lius: Translation Model Based Instructional Lingustic Using Continual Instruction Tuning In Kupang Malaycited by 0
  9. 2026Who Bridges Safety? Identifying and Targeting Cross-Lingual Shared Safety Pathwayscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Towards Alignment-Centric Paradigm: A Survey of Instruction Tuning in Large Language Models
  2. 2024X-Instruction: Aligning Language Model in Low-resource Languages with Self-curated Cross-lingual Instructions
  3. 2023TRACE: A Comprehensive Benchmark for Continual Learning in Large Language Models
  4. 2024CrossIn: An Efficient Instruction Tuning Approach for Cross-Lingual Knowledge Alignment
  5. 2024M2Lingual: Enhancing Multilingual, Multi-Turn Instruction Alignment in Large Language Models
  6. 2023Extrapolating Large Language Models to Non-English by Aligning Languages
  7. 2023BayLing: Bridging Cross-lingual Alignment and Instruction Following through Interactive Translation for Large Language Models
  8. 2023The Unlocking Spell on Base LLMs: Rethinking Alignment via In-Context Learning