mlsanityData to 7 Oct 2026

RLHF Can Speak Many Languages: Unlocking Multilingual Preference Optimization for LLMs

arXiv 2407.02552 ↗First posted 2 Jul 2024cs.CL cs.AI cs.LGcited by 42

On the map: LLM alignment / RLHF / preference optimisation → LLM preference optimization methods

Abstract (arXiv)

Preference optimization techniques have become a standard final stage for training state-of-art large language models (LLMs). However, despite widespread adoption, the vast majority of work to-date has focused on first-class citizen languages like English and Chinese. This captures a small fraction of the languages in the world, but also makes it unclear which aspects of current state-of-the-art research transfer to a multilingual setting. In this work, we perform an exhaustive study to achieve a new state-of-the-art in aligning multilingual LLMs. We introduce a novel, scalable method for generating high-quality multilingual feedback data to balance data coverage. We establish the benefits of cross-lingual transfer and increased dataset size in preference training. Our preference-trained model achieves a 54.4% win-rate against Aya 23 8B, the current state-of-the-art multilingual LLM in its parameter class, and a 69.5% win-rate or higher against widely used models like Gemma-1.1-7B-it, Llama-3-8B-Instruct, Mistral-7B-Instruct-v0.3. As a result of our study, we expand the frontier of alignment techniques to 23 languages covering half of the world's population.

Key facts. Cited by 42 papers in the map (data through 7 Oct 2026). Placed in LLM preference optimization methods, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 54 of its 86 references that are in the map. First posted on arXiv 2 Jul 2024.

Builds on

54 of 86 references.

Closest ancestors · 8

Same direction, newest first

  1. 2024Understanding the performance gap between online and offline alignment algorithms14 May
  2. 2024Advancing Translation Preference Modeling with RLHF: A Step Towards Cost-Effective Solution18 Feb
  3. 2023A General Theoretical Paradigm to Understand Learning from Human Preferences18 Oct
  4. 2023Group Preference Optimization: Few-Shot Alignment of Large Language Models17 Oct
  5. 2023Verbosity Bias in Preference Labeling by Large Language Models16 Oct
  6. 2023Okapi: Instruction-tuned Large Language Models in Multiple Languages with Reinforcement Learning from Human Feedback29 Jul
  7. 2022Training language models to follow instructions with human feedback4 Mar
  8. 2021A General Language Assistant as a Laboratory for Alignment1 Dec

Other references · 46

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2017Proximal Policy Optimization Algorithmscited by 14,441
  3. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  4. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,916
  5. 2020Measuring Massive Multitask Language Understandingcited by 7,768
  6. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,018
  7. 2017Deep reinforcement learning from human preferencescited by 3,846
  8. 2023Let's Verify Step by Stepcited by 3,598
  9. 2024Gemini 1.5: Unlocking multimodal understanding across millions of tokens of contextcited by 3,175
  10. 2020Learning to summarize from human feedbackcited by 2,562
  11. 2021WebGPT: Browser-assisted question-answering with human feedbackcited by 1,722
  12. 2022No Language Left Behind: Scaling Human-Centered Machine Translationcited by 1,290
34 more
  1. 2019ZeRO: Memory Optimizations Toward Training Trillion Parameter Modelscited by 1,278
  2. 2022Scaling Laws for Reward Model Overoptimizationcited by 1,056
  3. 2022Language Models are Multilingual Chain-of-Thought Reasonerscited by 518
  4. 2021Documenting Large Webtext Corpora: A Case Study on the Colossal Clean Crawled Corpuscited by 481
  5. 2023Scalable Extraction of Training Data from (Production) Language Modelscited by 415
  6. 2023Understanding the Effects of RLHF on LLM Generalisation and Diversitycited by 386
  7. 2023Toxicity in ChatGPT: Analyzing Persona-assigned Language Modelscited by 356
  8. 2021XL-Sum: Large-Scale Multilingual Abstractive Summarization for 44 Languagescited by 346
  9. 2020XCOPA: A Multilingual Dataset for Causal Commonsense Reasoningcited by 330
  10. 2024Replacing Judges with Juries: Evaluating LLM Generations with a Panel of Diverse Modelscited by 301
  11. 2021Few-shot Learning with Multilingual Language Modelscited by 278
  12. 2023Analyzing Leakage of Personally Identifiable Information in Language Modelscited by 278
  13. 2018Characterizing and Avoiding Negative Transfercited by 226
  14. 2023Multilingual Jailbreak Challenges in Large Language Modelscited by 222
  15. 2023Gender bias and stereotypes in Large Language Modelscited by 185
  16. 2024Preference Fine-Tuning of LLMs Should Leverage Suboptimal, On-Policy Datacited by 184
  17. 2024Aya Dataset: An Open-Access Collection for Multilingual Instruction Tuningcited by 174
  18. 2023Increasing Diversity While Maintaining Accuracy: Text Data Generation with Large Language Models and Human Interventionscited by 121
  19. 2024Aya 23: Open Weight Releases to Further Multilingual Progresscited by 121
  20. 2023Aligning Language Models to User Opinionscited by 98
  21. 2024RLHF Deciphered: A Critical Analysis of Reinforcement Learning from Human Feedback for LLMscited by 89
  22. 2021What's in the Box? A Preliminary Analysis of Undesirable Content in the Common Crawl Corpuscited by 79
  23. 2023The Data Provenance Initiative: A Large Scale Audit of Dataset Licensing & Attribution in AIcited by 73
  24. 2021Machine Translationese: Effects of Algorithmic Bias on Linguistic Complexity in Machine Translationcited by 70
  25. 2023Privacy in Large Language Models: Attacks, Defenses and Future Directionscited by 48
  26. 2023Improving Diversity of Demographic Representation in Large Language Models via Collective-Critiques and Self-Votingcited by 46
  27. 2023GPTAraEval: A Comprehensive Evaluation of ChatGPT on Arabic NLPcited by 37
  28. 2023On Evaluating and Mitigating Gender Biases in Multilingual Settingscited by 31
  29. 2022What a Creole Wants, What a Creole Needscited by 27
  30. 2024Reuse Your Rewards: Reward Model Transfer for Zero-Shot Cross-Lingual Alignmentcited by 26
  31. 2023Peering Through Preferences: Unraveling Feedback Acquisition for Aligning Large Language Modelscited by 23
  32. 2023Which Prompts Make The Difference? Data Prioritization For Efficient Human LLM Evaluationcited by 18
  33. 2023Indian-BhED: A Dataset for Measuring India-Centric Biases in Large Language Modelscited by 17
  34. 2023A Survey of Corpora for Germanic Low-Resource Languages and Dialectscited by 16

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 12; 2025: 22; 2026: 8.

42 papers.

Sort:

By area:

  1. 2024Does Refusal Training in LLMs Generalize to the Past Tense?cited by 74
  2. 2024M-RewardBench: Evaluating Reward Models in Multilingual Settingscited by 50
  3. 2024To Code, or Not To Code? Exploring Impact of Code in Pre-trainingcited by 47
  4. 2024MM-Eval: A Multilingual Meta-Evaluation Benchmark for LLM-as-a-Judge and Reward Modelscited by 34
  5. 2024Language Imbalance Driven Rewarding for Multilingual Self-improvingcited by 27
  6. 2024Preference Tuning with Human Feedback on Language, Speech, and Vision Tasks: A Surveycited by 23
  7. 2025Learn Globally, Speak Locally: Bridging the Gaps in Multilingual Reasoningcited by 23
  8. 2024Layer Swapping for Zero-Shot Cross-Lingual Transfer in Large Language Modelscited by 20
  9. 2024On the Limitations of Compute Thresholds as a Governance Strategycited by 17
  10. 2024Cross-lingual Transfer of Reward Models in Multilingual Alignmentcited by 17
  11. 2025Reasoning Beyond Limits: Advances and Open Problems for LLMscited by 16
  12. 2025Cultural Learning-Based Culture Adaptation of Language Modelscited by 16
30 more
  1. 2025Krikri: Advancing Open Large Language Models for Greekcited by 13
  2. 2025Improving Multilingual Language Models by Aligning Representations through Steeringcited by 11
  3. 2024The Future of Open Human Feedbackcited by 10
  4. 2025The Multilingual Divide and Its Impact on Global AI Safetycited by 10
  5. 2025Aligning LLMs for Multilingual Consistency in Enterprise Applicationscited by 10
  6. 2025Implicit Cross-Lingual Rewarding for Efficient Multilingual Preference Alignmentcited by 9
  7. 2025MT-RewardTree: A Comprehensive Framework for Advancing LLM-Based Machine Translation via Reward Modelingcited by 9
  8. 2025When Life Gives You Samples: The Benefits of Scaling up Inference Compute for Multilingual LLMscited by 9
  9. 2025Bridging the Linguistic Divide: A Survey on Leveraging Large Language Models for Machine Translationcited by 8
  10. 2024Multilingual Arbitrage: Optimizing Data Pools to Accelerate Multilingual Progresscited by 6
  11. 2025The Hidden Space of Safety: Understanding Preference-Tuned LLMs in Multilingual contextcited by 6
  12. 2025Redefining Machine Translation on Social Network Services with Large Language Modelscited by 6
  13. 2024Cross-Lingual Transfer of Debiasing and Detoxification in Multilingual LLMs: An Extensive Investigationcited by 5
  14. 2025Exploring Persona Sentiment Sensitivity in Personalized Dialogue Generationcited by 3
  15. 2025Got Compute, but No Data: Lessons From Post-training a Finnish LLMcited by 3
  16. 2025CAPO: Confidence Aware Preference Optimization Learning for Multilingual Preferencescited by 3
  17. 2026Mending the Holes: Mitigating Reward Hacking in Reinforcement Learning for Multilingual Translationcited by 3
  18. 2025CONGRAD:Conflicting Gradient Filtering for Multilingual Preference Alignmentcited by 2
  19. 2026Structuring the Space of Sociotechnical Alignmentcited by 2
  20. 2025The Effect of Language Diversity When Fine-Tuning Large Language Models for Translationcited by 1
  21. 2025IndoPref: A Multi-Domain Pairwise Preference Dataset for Indonesiancited by 1
  22. 2025CM-Align: Consistency-based Multilingual Alignment for Large Language Modelscited by 1
  23. 2026The Illusion of Cross-Lingual Safety in Low-Resource Languagescited by 1
  24. 2025Rethinking what Matters: Effective and Robust Multilingual Realignment for Low-Resource Languagescited by 0
  25. 2025Fluent Alignment with Disfluent Judges: Post-training for Lower-resource Languagescited by 0
  26. 2026CroCo: Cross-Lingual Contrastive Preference Tuning on Self-Generationscited by 0
  27. 2026A Survey of Toxicity Detection and Mitigation Strategies for Multilingual Language Modelscited by 0
  28. 2026Strengthening Target-Language Features: SAE-Based Steering for Multilingual Inferencecited by 0
  29. 2026Language Chain in Alignment: Cross-lingual Ranking Preference Optimizationcited by 0
  30. 2026Pak3H: Evaluating the Cost of Cultural Mismatch in LLM Alignment with a Human-Contextualized Urdu Benchmarkcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Towards a Unified View of Preference Learning for Large Language Models: A Survey
  2. 2024Optimizing LLMs with Direct Preferences: A Data Efficiency Perspective
  3. 2026Meta-Learning Preferences for Multilingual LLM Alignment
  4. 2024Multi-Response Preference Optimization with Augmented Ranking Dataset
  5. 2024A Survey on Human Preference Learning for Large Language Models
  6. 2024Investigating on RLHF methodology
  7. 2024LLMs Beyond English: Scaling the Multilingual Capability of LLMs with Cross-Lingual Feedback
  8. 2024Unified Preference Optimization: Language Model Alignment Beyond the Preference Frontier