mlsanityData to 14 Oct 2026

Empowering Cross-lingual Abilities of Instruction-tuned Large Language Models by Translation-following demonstrations

arXiv 2308.14186 ↗First posted 27 Aug 2023cs.CL cs.AIcited by 35

Not on the map. Why?

Abstract (arXiv)

The language ability of Large Language Models (LLMs) is often unbalanced towards English because of the imbalance in the distribution of the pre-training data. This disparity is demanded in further fine-tuning and affecting the cross-lingual abilities of LLMs. In this paper, we propose to empower Instructiontuned LLMs (It-LLMs) in languages other than English by building semantic alignment between them. Hence, we propose CrossAlpaca, an It-LLM with cross-lingual instruction-following and Translation-following demonstrations to improve semantic alignment between languages. We validate our approach on the multilingual Question Answering (QA) benchmarks XQUAD and MLQA and adapted versions of MMLU and BBH. Our models, tested over six different languages, outperform the It-LLMs tuned on monolingual data. The final results show that instruction tuning on non-English data is not enough and that semantic alignment can be further improved by Translation-following demonstrations.

Key facts. Cited by 35 papers in the map (data through 14 Oct 2026). Builds on 37 of its 47 references that are in the map. First posted on arXiv 27 Aug 2023.

Builds on

37 of 47 references.

References · 37

  1. 2020Language Models are Few-Shot Learnerscited by 32,419
  2. 2022Training language models to follow instructions with human feedbackcited by 15,971
  3. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
  4. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  5. 2020Measuring Massive Multitask Language Understandingcited by 7,963
  6. 2016SQuAD: 100,000+ Questions for Machine Comprehension of Textcited by 4,148
  7. 2021Prefix-Tuning: Optimizing Continuous Prompts for Generationcited by 3,915
  8. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,780
  9. 2022Self-Instruct: Aligning Language Models with Self-Generated Instructionscited by 2,724
  10. 2020The Pile: An 800GB Dataset of Diverse Text for Language Modelingcited by 2,411
  11. 2019Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelismcited by 1,868
  12. 2022Challenging BIG-Bench Tasks and Whether Chain-of-Thought Can Solve Themcited by 1,807
25 more
  1. 2019BERT Rediscovers the Classical NLP Pipelinecited by 1,059
  2. 2015Aligning Books and Movies: Towards Story-like Visual Explanations by Watching Movies and Reading Bookscited by 1,043
  3. 2023A Multitask, Multilingual, Multimodal Evaluation of ChatGPT on Reasoning, Hallucination, and Interactivitycited by 894
  4. 2023The RefinedWeb Dataset for Falcon LLM: Outperforming Curated Corpora with Web Data, and Web Data Onlycited by 763
  5. 2019On the Cross-lingual Transferability of Monolingual Representationscited by 578
  6. 2021Documenting Large Webtext Corpora: A Case Study on the Colossal Clean Crawled Corpuscited by 492
  7. 2019MLQA: Evaluating Cross-lingual Extractive Question Answeringcited by 340
  8. 2023Efficient and Effective Text Encoding for Chinese LLaMA and Alpacacited by 305
  9. 2023Not All Languages Are Created Equal in LLMs: Improving Multilingual Capability by Cross-Lingual-Thought Promptingcited by 165
  10. 2023Multilingual Machine Translation with Large Language Models: Empirical Results and Analysiscited by 160
  11. 2022Towards a Cleaner Document-Oriented Multilingual Crawled Corpuscited by 143
  12. 2023Glot500: Scaling Multilingual Corpora and Language Models to 500 Languagescited by 116
  13. 2022Language Contamination Helps Explain the Cross-lingual Capabilities of English Pretrained Modelscited by 92
  14. 2023Extrapolating Large Language Models to Non-English by Aligning Languagescited by 69
  15. 2020Pre-training Multilingual Neural Machine Translation by Leveraging Alignment Informationcited by 64
  16. 2023BigTranslate: Augmenting Large Language Models with Multilingual Translation Capability over 100 Languagescited by 60
  17. 2023Adaptive Machine Translation with Large Language Modelscited by 58
  18. 2023Eliciting the Translation Ability of Large Language Models via Multilingual Finetuning with Translation Instructionscited by 56
  19. 2021Lightweight Adapter Tuning for Multilingual Speech Translationcited by 54
  20. 2023TIM: Teaching Large Language Models to Translate with Comparisoncited by 50
  21. 2023BayLing: Bridging Cross-lingual Alignment and Instruction Following through Interactive Translation for Large Language Modelscited by 42
  22. 2013Complexity of Word Collocation Networks: A Preliminary Structural Analysiscited by 17
  23. 2023Camoscio: an Italian Instruction-tuned LLaMAcited by 13
  24. 2021On the Language-specificity of Multilingual BERT and the Impact of Fine-tuningcited by 8
  25. 2023Exploring Linguistic Properties of Monolingual BERTs with Typological Classification among Languagescited by 6

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 5; 2024: 15; 2025: 8; 2026: 7.

35 papers.

Sort:

By area:

  1. 2024Continual Pre-Training for Cross-Lingual LLM Adaptation: Enhancing Japanese Language Capabilitiescited by 111
  2. 2023Monolingual or Multilingual Instruction Tuning: Which Makes a Better Alpacacited by 60
  3. 2024Multilingual Large Language Model: A Survey of Resources, Taxonomy and Frontierscited by 51
  4. 2023PLUG: Leveraging Pivot Language in Cross-Lingual Instruction Tuningcited by 39
  5. 2023Improving In-context Learning of Multilingual Generative Language Models with Cross-lingual Alignmentcited by 33
  6. 2025Evaluating o1-Like LLMs: Unlocking Reasoning for Translation through Comprehensive Analysiscited by 26
  7. 2024Bridging the Language Gaps in Large Language Models with Inference-Time Cross-Lingual Interventioncited by 24
  8. 2024Teaching Llama a New Language Through Cross-Lingual Knowledge Transfercited by 23
  9. 2024Multilingual Large Language Models: A Systematic Surveycited by 18
  10. 2024How Reliable Are Automatic Evaluation Methods for Instruction-Tuned LLMs?cited by 13
  11. 2024Zero-shot cross-lingual transfer in instruction tuning of large language modelscited by 11
  12. 2025GLaD: Geometric Latent Distillation for Vision-Language-Action Modelscited by 11
23 more
  1. 2025Rethinking Multilingual Continual Pretraining: Data Mixing for Adapting LLMs Across Languages and Resourcescited by 10
  2. 2024Crosslingual Capabilities and Knowledge Barriers in Multilingual Large Language Modelscited by 9
  3. 2023Empowering Multi-step Reasoning across Languages via Tree-of-Thoughtscited by 8
  4. 2024Facilitating large language model Russian adaptation with Learned Embedding Propagationcited by 8
  5. 2024M2Lingual: Enhancing Multilingual, Multi-Turn Instruction Alignment in Large Language Modelscited by 7
  6. 2024LLM-based Translation Inference with Iterative Bilingual Understandingcited by 6
  7. 2024Prompting with Phonemes: Enhancing LLMs' Multilinguality for Non-Latin Script Languagescited by 6
  8. 2024Make Imagination Clearer! Stable Diffusion-based Visual Imagination for Multimodal Machine Translationcited by 5
  9. 2025Improving Multilingual Retrieval-Augmented Language Models through Dialectic Reasoning Argumentationscited by 5
  10. 2025Understanding LLMs' Cross-Lingual Context Retrieval: How Good It Is And Where It Comes Fromcited by 4
  11. 2025Rethinking Cross-lingual Gaps from a Statistical Viewpointcited by 4
  12. 2024Fine-tuning with HED-IT: The impact of human post-editing for dialogical language modelscited by 3
  13. 2026OpenSeal: Good, Fast, and Cheap Construction of an Open-Source Southeast Asian LLM via Parallel Datacited by 3
  14. 2023HANS, are you clever? Clever Hans Effect Analysis of Neural Systemscited by 2
  15. 2026Mechanistic Circuit Identification for Controllable Data Generationcited by 2
  16. 2025CC-Tuning: A Cross-Lingual Connection Mechanism for Improving Joint Multilingual Supervised Fine-Tuningcited by 1
  17. 2026Disentangling Language Roles in Multilingual LLM Task Executioncited by 1
  18. 2024WangchanLion and WangchanX MRC Evalcited by 0
  19. 2025Prompt, Translate, Fine-Tune, Re-Initialize, or Instruction-Tune? Adapting LLMs for In-Context Learning in Low-Resource Languagescited by 0
  20. 2026Leveraging LLMs For Turkish Skill Extractioncited by 0
  21. 2026Language Chain in Alignment: Cross-lingual Ranking Preference Optimizationcited by 0
  22. 2026Recognized but Not Produced: A Generation Benchmark for Culturally Specific Kinship Termscited by 0
  23. 2026HexaBee-4B: Six-Language Translation with Multilingual Answering and Bidirectional Supervisioncited by 0