mlsanityData to 7 Oct 2026

Okapi: Instruction-tuned Large Language Models in Multiple Languages with Reinforcement Learning from Human Feedback

arXiv 2307.16039 ↗First posted 29 Jul 2023cs.CL cs.LGcited by 189

On the map: LLM alignment / RLHF / preference optimisation → LLM preference optimization methods

Abstract (arXiv)

A key technology for the development of large language models (LLMs) involves instruction tuning that helps align the models' responses with human expectations to realize impressive learning abilities. Two major approaches for instruction tuning characterize supervised fine-tuning (SFT) and reinforcement learning from human feedback (RLHF), which are currently applied to produce the best commercial LLMs (e.g., ChatGPT). To improve the accessibility of LLMs for research and development efforts, various instruction-tuned open-source LLMs have also been introduced recently, e.g., Alpaca, Vicuna, to name a few. However, existing open-source LLMs have only been instruction-tuned for English and a few popular languages, thus hindering their impacts and accessibility to many other languages in the world. Among a few very recent work to explore instruction tuning for LLMs in multiple languages, SFT has been used as the only approach to instruction-tune LLMs for multiple languages. This has left a significant gap for fine-tuned LLMs based on RLHF in diverse languages and raised important questions on how RLHF can boost the performance of multilingual instruction tuning. To overcome this issue, we present Okapi, the first system with instruction-tuned LLMs based on RLHF for multiple languages. Okapi introduces instruction and response-ranked data in 26 diverse languages to facilitate the experiments and development of future multilingual LLM research. We also present benchmark datasets to enable the evaluation of generative LLMs in multiple languages. Our experiments demonstrate the advantages of RLHF for multilingual instruction over SFT for different base models and datasets. Our framework and resources are released at https://github.com/nlp-uoregon/Okapi.

Key facts. Cited by 189 papers in the map (data through 7 Oct 2026). Placed in LLM preference optimization methods, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 55 of its 77 references that are in the map. First posted on arXiv 29 Jul 2023.

Builds on

55 of 77 references.

Closest ancestors · 2

Same direction, newest first

  1. 2022Self-Instruct: Aligning Language Models with Self-Generated Instructions20 Dec
  2. 2022Training language models to follow instructions with human feedback4 Mar

Other references · 53

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2020Language Models are Few-Shot Learnerscited by 32,199
  4. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  5. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  6. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  7. 2020Measuring Massive Multitask Language Understandingcited by 7,768
  8. 2019BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehensioncited by 5,874
  9. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,710
  10. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,018
  11. 2021On the Opportunities and Risks of Foundation Modelscited by 3,961
  12. 2019Unsupervised Cross-lingual Representation Learning at Scalecited by 3,907
38 more
  1. 2017Deep reinforcement learning from human preferencescited by 3,846
  2. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,747
  3. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,464
  4. 2022Scaling Instruction-Finetuned Language Modelscited by 2,997
  5. 2021TruthfulQA: Measuring How Models Mimic Human Falsehoodscited by 2,919
  6. 2020Learning to summarize from human feedbackcited by 2,562
  7. 2022Emergent Abilities of Large Language Modelscited by 2,483
  8. 2022BLOOM: A 176B-Parameter Open-Access Multilingual Language Modelcited by 1,968
  9. 2020mT5: A massively multilingual pre-trained text-to-text transformercited by 1,813
  10. 2021Multitask Prompted Training Enables Zero-Shot Task Generalizationcited by 1,462
  11. 2021Scaling Language Models: Methods, Analysis & Insights from Training Gophercited by 1,199
  12. 2020Multilingual Denoising Pre-training for Neural Machine Translationcited by 1,028
  13. 2021Ethical and social risks of harm from Language Modelscited by 957
  14. 2018XNLI: Evaluating Cross-lingual Sentence Representationscited by 913
  15. 2023A Multitask, Multilingual, Multimodal Evaluation of ChatGPT on Reasoning, Hallucination, and Interactivitycited by 888
  16. 2020XTREME: A Massively Multilingual Multi-task Benchmark for Evaluating Cross-lingual Generalizationcited by 738
  17. 2023The Flan Collection: Designing Data and Methods for Effective Instruction Tuningcited by 734
  18. 2019How multilingual is Multilingual BERT?cited by 642
  19. 2019On the Cross-lingual Transferability of Monolingual Representationscited by 575
  20. 2023How Close is ChatGPT to Human Experts? Comparison Corpus, Evaluation, and Detectioncited by 472
  21. 2021Recent Advances in Natural Language Processing via Large Pre-Trained Language Models: A Surveycited by 457
  22. 2019CamemBERT: a Tasty French Language Modelcited by 368
  23. 2019Beto, Bentz, Becas: The Surprising Cross-Lingual Effectiveness of BERTcited by 301
  24. 2020XGLUE: A New Benchmark Dataset for Cross-lingual Pre-training, Understanding and Generationcited by 221
  25. 2023ChatGPT Beyond English: Towards a Comprehensive Evaluation of Large Language Models in Multilingual Learningcited by 203
  26. 2019Cross-Lingual Ability of Multilingual BERT: An Empirical Studycited by 182
  27. 2023Is ChatGPT A Good Translator? Yes With GPT-4 As The Enginecited by 172
  28. 2021Alignment of Language Agentscited by 167
  29. 2023A Categorical Archive of ChatGPT Failurescited by 167
  30. 2021Understanding the Capabilities, Limitations, and Societal Impact of Large Language Modelscited by 159
  31. 2023LaMini-LM: A Diverse Herd of Distilled Models from Large-Scale Instructionscited by 124
  32. 2021Mr. TyDi: A Multi-lingual Benchmark for Dense Retrievalcited by 118
  33. 2019Unicoder: A Universal Language Encoder by Pre-training with Multiple Cross-lingual Taskscited by 96
  34. 201975 Languages, 1 Model: Parsing Universal Dependencies Universallycited by 90
  35. 2023Bactrian-X: Multilingual Replicable Instruction-Following Models with Low-Rank Adaptationcited by 79
  36. 2021Trankit: A Light-Weight Transformer-based Toolkit for Multilingual Natural Language Processingcited by 68
  37. 2020BARThez: a Skilled Pretrained French Sequence-to-Sequence Modelcited by 39
  38. 2022Models and Datasets for Cross-Lingual Summarisationcited by 36

50 most cited of 53 All 53 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 2; 2024: 84; 2025: 68; 2026: 35.

189 papers.

Sort:

By area:

  1. 2024Aya Model: An Instruction Finetuned Open-Access Multilingual Language Modelcited by 292
  2. 2024Aya Dataset: An Open-Access Collection for Multilingual Instruction Tuningcited by 174
  3. 2024Aya 23: Open Weight Releases to Further Multilingual Progresscited by 121
  4. 2024EuroLLM: Multilingual Language Models for Europecited by 103
  5. 2024KMMLU: Measuring Massive Multitask Language Understanding in Koreancited by 91
  6. 2024Stable LM 2 1.6B Technical Reportcited by 86
  7. 2024INCLUDE: Evaluating Multilingual Language Understanding with Regional Knowledgecited by 78
  8. 2024Understanding and Mitigating Language Confusion in LLMscited by 75
  9. 2024Multilingual Instruction Tuning With Just a Pinch of Multilingualitycited by 71
  10. 2024The Multilingual Alignment Prism: Aligning Global and Local Preferences to Reduce Harmcited by 67
  11. 2024LLMs Beyond English: Scaling the Multilingual Capability of LLMs with Cross-Lingual Feedbackcited by 51
  12. 2024M-RewardBench: Evaluating Reward Models in Multilingual Settingscited by 50
38 more
  1. 2024Multilingual Large Language Model: A Survey of Resources, Taxonomy and Frontierscited by 49
  2. 2024Jamba-1.5: Hybrid Transformer-Mamba Models at Scalecited by 48
  3. 2024AraDiCE: Benchmarks for Dialectal and Cultural Capabilities in LLMscited by 46
  4. 2024IrokoBench: A New Benchmark for African Languages in the Age of Large Language Modelscited by 43
  5. 2024RLHF Can Speak Many Languages: Unlocking Multilingual Preference Optimization for LLMscited by 42
  6. 2024SEACrowd: A Multilingual Multimodal Data Hub and Benchmark Suite for Southeast Asian Languagescited by 41
  7. 2024How Does Quantization Affect Multilingual LLMs?cited by 41
  8. 2023Turning English-centric LLMs Into Polyglots: How Much Multilinguality Is Needed?cited by 39
  9. 2024LoRA-Flow: Dynamic LoRA Fusion for Large Language Models in Generative Taskscited by 39
  10. 2024Towards Evaluating and Building Versatile Large Language Models for Medicinecited by 38
  11. 2024MM-Eval: A Multilingual Meta-Evaluation Benchmark for LLM-as-a-Judge and Reward Modelscited by 34
  12. 2024Towards Understanding the Fragility of Multilingual LLMs against Fine-Tuning Attackscited by 33
  13. 2025Linguistic Generalizability of Test-Time Scaling in Mathematical Reasoningcited by 33
  14. 2024Preference Tuning For Toxicity Mitigation Generalizes Across Languagescited by 31
  15. 2024CaLMQA: Exploring culturally specific long-form question answering across 23 languagescited by 31
  16. 2024TurkishMMLU: Measuring Massive Multitask Language Understanding in Turkishcited by 31
  17. 2025A Survey of Multilingual Reasoning in Language Modelscited by 31
  18. 2025The Bitter Lesson Learned from 2,000+ Multilingual Benchmarkscited by 31
  19. 2024Zero-Shot Tokenizer Transfercited by 30
  20. 2024NativQA: Multilingual Culturally-Aligned Natural Query for LLMscited by 30
  21. 2025MergeBench: A Benchmark for Merging Domain-Specialized LLMscited by 30
  22. 2024RomanSetu: Efficiently unlocking multilingual capabilities of Large Language Models via Romanizationcited by 29
  23. 2024P-MMEval: A Parallel Multilingual Multitask Benchmark for Consistent Evaluation of LLMscited by 29
  24. 2024Evaluating the Elementary Multilingual Capabilities of Large Language Models with MultiQcited by 27
  25. 2024Language Imbalance Driven Rewarding for Multilingual Self-improvingcited by 27
  26. 2024Reuse Your Rewards: Reward Model Transfer for Zero-Shot Cross-Lingual Alignmentcited by 26
  27. 2024Culturally Aware and Adapted NLP: A Taxonomy and a Survey of the State of the Artcited by 26
  28. 2024MEXA: Multilingual Evaluation of English-Centric LLMs via Cross-Lingual Alignmentcited by 25
  29. 2024Bridging the Language Gaps in Large Language Models with Inference-Time Cross-Lingual Interventioncited by 24
  30. 2025BenchMAX: A Comprehensive Multilingual Evaluation Suite for Large Language Modelscited by 23
  31. 2024LangBridge: Multilingual Reasoning Without Multilingual Supervisioncited by 22
  32. 2024Scaling Laws for Multilingual Language Modelscited by 22
  33. 2025BOUQuET: dataset, Benchmark and Open initiative for Universal Quality Evaluation in Translationcited by 21
  34. 2024Spanish and LLM Benchmarks: is MMLU Lost in Translation?cited by 20
  35. 2025Language-Specific Latent Process Hinders Cross-Lingual Performancecited by 20
  36. 2025The Emergence of Abstract Thought in Large Language Models Beyond Any Languagecited by 20
  37. 2025Exploring and Mitigating Adversarial Manipulation of Voting-Based Leaderboardscited by 19
  38. 2025Localized Cultural Knowledge is Conserved and Controllable in Large Language Modelscited by 19

50 most cited of 189 All 189 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024Minor SFT loss for LLM fine-tune to increase performance and reduce model deviation
  2. 2023Understanding the Effects of RLHF on LLM Generalisation and Diversity
  3. 2024Non-instructional Fine-tuning: Enabling Instruction-Following Capabilities in Pre-trained Language Models without Instruction-Following Data
  4. 2023An Empirical Study of Instruction-tuning Large Language Models in Chinese
  5. 2023Tuna: Instruction Tuning using Feedback from Large Language Models
  6. 2024A Framework for Fine-Tuning LLMs using Heterogeneous Feedback
  7. 2025Massive Supervised Fine-tuning Experiments Reveal How Data, Layer, and Training Factors Shape LLM Alignment Quality
  8. 2023Reflection-Tuning: Data Recycling Improves LLM Instruction-Tuning