mlsanityData to 14 Oct 2026

Reuse Your Rewards: Reward Model Transfer for Zero-Shot Cross-Lingual Alignment

arXiv 2404.12318 ↗First posted 18 Apr 2024cs.CLcited by 26

On the map: LLM alignment / RLHF / preference optimisation → Reward modeling and RLHF

Abstract (arXiv)

Aligning language models (LMs) based on human-annotated preference data is a crucial step in obtaining practical and performant LM-based systems. However, multilingual human preference data are difficult to obtain at scale, making it challenging to extend this framework to diverse languages. In this work, we evaluate a simple approach for zero-shot cross-lingual alignment, where a reward model is trained on preference data in one source language and directly applied to other target languages. On summarization and open-ended dialog generation, we show that this method is consistently successful under comprehensive evaluation settings, including human evaluation: cross-lingually aligned models are preferred by humans over unaligned models on up to >70% of evaluation instances. We moreover find that a different-language reward model sometimes yields better aligned models than a same-language reward model. We also identify best practices when there is no language-specific data for even supervised finetuning, another component in alignment.

Key facts. Cited by 26 papers in the map (data through 14 Oct 2026). Placed in Reward modeling and RLHF, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 55 of its 76 references that are in the map. First posted on arXiv 18 Apr 2024.

Builds on

55 of 76 references.

Closest ancestors · 7

Same direction, newest first

  1. 2023A Long Way to Go: Investigating Length Correlations in RLHF5 Oct
  2. 2023Reward Model Ensembles Help Mitigate Overoptimization4 Oct
  3. 2023Fine-Grained Human Feedback Gives Better Rewards for Language Model Training2 Jun
  4. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Model29 May
  5. 2022Scaling Laws for Reward Model Overoptimization19 Oct
  6. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback12 Apr
  7. 2019Fine-Tuning Language Models from Human Preferences18 Sep

Other references · 48

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  2. 2022Training language models to follow instructions with human feedbackcited by 15,971
  3. 2017Proximal Policy Optimization Algorithmscited by 15,215
  4. 2019Unsupervised Cross-lingual Representation Learning at Scalecited by 3,935
  5. 2022Self-Instruct: Aligning Language Models with Self-Generated Instructionscited by 2,724
  6. 2020Learning to summarize from human feedbackcited by 2,588
  7. 2020mT5: A massively multilingual pre-trained text-to-text transformercited by 1,818
  8. 2018Don't Give Me the Details, Just the Summary! Topic-Aware Convolutional Neural Networks for Extreme Summarizationcited by 1,317
  9. 2019Cross-lingual Language Model Pretrainingcited by 1,090
  10. 2023Large Language Models are not Fair Evaluatorscited by 1,085
  11. 2018XNLI: Evaluating Cross-lingual Sentence Representationscited by 915
  12. 2023A General Theoretical Paradigm to Understand Learning from Human Preferencescited by 912
36 more
  1. 2018Adafactor: Adaptive Learning Rates with Sublinear Memory Costcited by 905
  2. 2020The State and Fate of Linguistic Diversity and Inclusion in the NLP Worldcited by 798
  3. 2023AlpacaFarm: A Simulation Framework for Methods that Learn from Human Feedbackcited by 748
  4. 2020XTREME: A Massively Multilingual Multi-task Benchmark for Evaluating Cross-lingual Generalizationcited by 742
  5. 2023OpenAssistant Conversations -- Democratizing Large Language Model Alignmentcited by 721
  6. 2019How multilingual is Multilingual BERT?cited by 645
  7. 2023Large Language Models Are Not Robust Multiple Choice Selectorscited by 486
  8. 2023How Far Can Camels Go? Exploring the State of Instruction Tuning on Open Resourcescited by 408
  9. 2023Learning to Compress Prompts with Gist Tokenscited by 390
  10. 2021XL-Sum: Large-Scale Multilingual Abstractive Summarization for 44 Languagescited by 347
  11. 2019Beto, Bentz, Becas: The Surprising Cross-Lingual Effectiveness of BERTcited by 301
  12. 2023MEGA: Multilingual Evaluation of Generative AIcited by 293
  13. 2021The GEM Benchmark: Natural Language Generation, its Evaluation and Metricscited by 239
  14. 2023L-Eval: Instituting Standardized Evaluation for Long Context Language Modelscited by 233
  15. 2022Challenges and Strategies in Cross-Cultural NLPcited by 218
  16. 2020Are All Languages Created Equal in Multilingual BERT?cited by 207
  17. 2023Okapi: Instruction-tuned Large Language Models in Multiple Languages with Reinforcement Learning from Human Feedbackcited by 190
  18. 2019Cross-Lingual Ability of Multilingual BERT: An Empirical Studycited by 182
  19. 2019Emerging Cross-lingual Structure in Pretrained Language Modelscited by 181
  20. 2024Aya Dataset: An Open-Access Collection for Multilingual Instruction Tuningcited by 177
  21. 2020WikiLingua: A New Benchmark Dataset for Cross-Lingual Abstractive Summarizationcited by 154
  22. 2020When Being Unseen from mBERT is just the Beginning: Handling New Languages With Multilingual Language Modelscited by 131
  23. 2024Theoretical guarantees on the best-of-n alignment policycited by 124
  24. 2020MLSUM: The Multilingual Summarization Corpuscited by 121
  25. 2023A Critical Evaluation of Evaluations for Long-form Question Answeringcited by 111
  26. 2023LongEval: Guidelines for Human Evaluation of Faithfulness in Long-form Summarizationcited by 96
  27. 2024The Language Barrier: Dissecting Safety Challenges of LLMs in Multilingual Contextscited by 94
  28. 2023CValues: Measuring the Values of Chinese Large Language Models from Safety to Responsibilitycited by 85
  29. 2023Human Feedback is not Gold Standardcited by 83
  30. 2024Multilingual Instruction Tuning With Just a Pinch of Multilingualitycited by 73
  31. 2023BUFFET: Benchmarking Large Language Models for Few-shot Cross-lingual Transfercited by 68
  32. 2023SEAHORSE: A Multilingual, Multifaceted Dataset for Summarization Evaluationcited by 42
  33. 2019Evaluating the Cross-Lingual Effectiveness of Massively Multilingual Neural Machine Translationcited by 27
  34. 2022Beyond Counting Datasets: A Survey of Multilingual Dataset Construction and Necessary Resourcescited by 25
  35. 2019Polyglot Contextual Representations Improve Crosslingual Transfercited by 16
  36. 2019A Critical Analysis of Biased Parsers in Unsupervised Parsingcited by 7

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 9; 2025: 13; 2026: 4.

26 papers.

Sort:

By area:

  1. 2024Tulu 3: Pushing Frontiers in Open Language Model Post-Trainingcited by 1,053
  2. 2024RLHF Can Speak Many Languages: Unlocking Multilingual Preference Optimization for LLMscited by 43
  3. 2024MM-Eval: A Multilingual Meta-Evaluation Benchmark for LLM-as-a-Judge and Reward Modelscited by 35
  4. 2025A Survey of Multilingual Reasoning in Language Modelscited by 33
  5. 2024LLM-as-a-Judge & Reward Model: What They Can and Cannot Docited by 32
  6. 2024Preference Tuning For Toxicity Mitigation Generalizes Across Languagescited by 31
  7. 2025reWordBench: Benchmarking and Improving the Robustness of Reward Models with Transformed Inputscited by 25
  8. 2024Multilingual Large Language Models: A Systematic Surveycited by 18
  9. 2025MPO: Multilingual Safety Alignment via Reward Gap Optimizationcited by 18
  10. 2024Cross-lingual Transfer of Reward Models in Multilingual Alignmentcited by 17
  11. 2025On the Robustness of Reward Models for Language Model Alignmentcited by 13
  12. 2024Crosslingual Capabilities and Knowledge Barriers in Multilingual Large Language Modelscited by 9
14 more
  1. 2025Implicit Cross-Lingual Rewarding for Efficient Multilingual Preference Alignmentcited by 9
  2. 2025When Life Gives You Samples: The Benefits of Scaling up Inference Compute for Multilingual LLMscited by 9
  3. 2025Demystifying Multilingual Chain-of-Thought in Process Reward Modelingcited by 6
  4. 2025Relic: Enhancing Reward Model Generalization for Low-Resource Indic Languages with Few-Shot Examplescited by 6
  5. 2025Can RLHF be More Efficient with Imperfect Reward Models? A Policy Coverage Perspectivecited by 4
  6. 2025MENLO: From Preferences to Proficiency -- Evaluating and Modeling Native-like Quality Across 47 Languagescited by 3
  7. 2025CAPO: Confidence Aware Preference Optimization Learning for Multilingual Preferencescited by 3
  8. 2025CM-Align: Consistency-based Multilingual Alignment for Large Language Modelscited by 2
  9. 2026Parallel-SFT: Improving Zero-Shot Cross-Programming-Language Transfer for Code RLcited by 2
  10. 2026Structuring the Space of Sociotechnical Alignmentcited by 2
  11. 2024Zero-shot Cross-Lingual Transfer for Synthetic Data Generation in Grammatical Error Detectioncited by 0
  12. 2025MTQ-Eval: Multilingual Text Quality Evaluation for Language Modelscited by 0
  13. 2026CroCo: Cross-Lingual Contrastive Preference Tuning on Self-Generationscited by 0
  14. 2026Language Chain in Alignment: Cross-lingual Ranking Preference Optimizationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2023On Diversified Preferences of Large Language Model Alignment
  2. 2024Elephant in the Room: Unveiling the Impact of Reward Model Quality in Alignment
  3. 2024Transforming and Combining Rewards for Aligning Large Language Models
  4. 2025Learning from Reference Answers: Versatile Language Model Alignment without Binary Human Preference Data
  5. 2026Meta-Learning Preferences for Multilingual LLM Alignment
  6. 2024RMB: Comprehensively Benchmarking Reward Models in LLM Alignment
  7. 2025Self-Consistency of the Internal Reward Models Improves Self-Rewarding Language Models
  8. 2024Rethinking the Role of Proxy Rewards in Language Model Alignment