mlsanityData to 14 Oct 2026

Cross-lingual Transfer of Reward Models in Multilingual Alignment

arXiv 2410.18027 ↗First posted 23 Oct 2024cs.CL cs.AIcited by 17

On the map: LLM alignment / RLHF / preference optimisation → Reward modeling and RLHF

Abstract (arXiv)

Reinforcement learning with human feedback (RLHF) is shown to largely benefit from precise reward models (RMs). However, recent studies in reward modeling schemes are skewed towards English, limiting the applicability of RLHF in multilingual alignments. In this work, we investigate the cross-lingual transfer of RMs trained in diverse languages, primarily from English. Our experimental results demonstrate the strong cross-lingual transfer of English RMs, exceeding target language RMs by 3~4% average increase in Multilingual RewardBench. Furthermore, we analyze the cross-lingual transfer of RMs through the representation shifts. Finally, we perform multilingual alignment to exemplify how cross-lingual transfer in RM propagates to enhanced multilingual instruction-following capability, along with extensive analyses on off-the-shelf RMs. We release the code, model, and data.

Key facts. Cited by 17 papers in the map (data through 14 Oct 2026). Placed in Reward modeling and RLHF, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 46 of its 62 references that are in the map. First posted on arXiv 23 Oct 2024.

Builds on

46 of 62 references.

Closest ancestors · 10

Same direction, newest first

  1. 2024Interpretable Preferences via Multi-Objective Reward Modeling and Mixture-of-Experts18 Jun
  2. 2024HelpSteer2: Open-source dataset for training top-performing reward models12 Jun
  3. 2024Reuse Your Rewards: Reward Model Transfer for Zero-Shot Cross-Lingual Alignment18 Apr
  4. 2024The N+ Implementation Details of RLHF with PPO: A Case Study on TL;DR Summarization24 Mar
  5. 2024RewardBench: Evaluating Reward Models for Language Modeling20 Mar
  6. 2024Secrets of RLHF in Large Language Models Part II: Reward Modeling11 Jan
  7. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Model29 May
  8. 2022Scaling Laws for Reward Model Overoptimization19 Oct
  9. 2019Fine-Tuning Language Models from Human Preferences18 Sep
  10. 2017Deep reinforcement learning from human preferences12 Jun

Other references · 36

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  2. 2024The Llama 3 Herd of Modelscited by 14,715
  3. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
  4. 2019Unsupervised Cross-lingual Representation Learning at Scalecited by 3,935
  5. 2023QLoRA: Efficient Finetuning of Quantized LLMscited by 3,418
  6. 2020Learning to summarize from human feedbackcited by 2,588
  7. 2024Qwen2 Technical Reportcited by 2,120
  8. 2020mT5: A massively multilingual pre-trained text-to-text transformercited by 1,818
  9. 2019ZeRO: Memory Optimizations Toward Training Trillion Parameter Modelscited by 1,301
  10. 2024The FineWeb Datasets: Decanting the Web for the Finest Text Data at Scalecited by 1,160
  11. 2024SimPO: Simple Preference Optimization with a Reference-Free Rewardcited by 963
  12. 2023BeaverTails: Towards Improved Safety Alignment of LLM via a Human-Preference Datasetcited by 860
24 more
  1. 2023Safe RLHF: Safe Reinforcement Learning from Human Feedbackcited by 656
  2. 2024OLMo: Accelerating the Science of Language Modelscited by 613
  3. 2024ORPO: Monolithic Preference Optimization without Reference Modelcited by 562
  4. 2024WildGuard: Open One-Stop Moderation Tools for Safety Risks, Jailbreaks, and Refusals of LLMscited by 496
  5. 2024Dolma: an Open Corpus of Three Trillion Tokens for Language Model Pretraining Researchcited by 483
  6. 2023UltraFeedback: Boosting Language Models with Scaled AI Feedbackcited by 406
  7. 20218-bit Optimizers via Block-wise Quantizationcited by 403
  8. 2024Magpie: Alignment Data Synthesis from Scratch by Prompting Aligned LLMs with Nothingcited by 346
  9. 2023The Belebele Benchmark: a Parallel Reading Comprehension Dataset in 122 Language Variantscited by 311
  10. 2024Aya Model: An Instruction Finetuned Open-Access Multilingual Language Modelcited by 295
  11. 2021Understanding Dataset Difficulty with V-Usable Informationcited by 291
  12. 2020Better Fine-Tuning by Reducing Representational Collapsecited by 157
  13. 2023HelpSteer: Multi-attribute Helpfulness Dataset for SteerLMcited by 142
  14. 2024Liger Kernel: Efficient Triton Kernels for LLM Trainingcited by 130
  15. 2021XLM-E: Cross-lingual Language Model Pre-training via ELECTRAcited by 72
  16. 2024Getting the most out of your tokenizer for pre-training and domain adaptationcited by 66
  17. 2024RLHF Can Speak Many Languages: Unlocking Multilingual Preference Optimization for LLMscited by 43
  18. 2023PLUG: Leveraging Pivot Language in Cross-Lingual Instruction Tuningcited by 39
  19. 2024Probing the Emergence of Cross-lingual Alignment during LLM Trainingcited by 39
  20. 2024X-ALMA: Plug & Play Modules and Adaptive Rejection for Quality Translation at Scalecited by 36
  21. 2019Pre-trained Language Model Representations for Language Generationcited by 34
  22. 2023Hyperpolyglot LLMs: Cross-Lingual Interpretability in Token Embeddingscited by 19
  23. 2022Representation Projection Invariance Mitigates Representation Collapsecited by 10
  24. 2024Stable Language Model Pre-training by Reducing Embedding Variabilitycited by 6

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 11; 2026: 6.

17 papers.

Sort:

By area:

  1. 2025Linguistic Generalizability of Test-Time Scaling in Mathematical Reasoningcited by 37
  2. 2025MPO: Multilingual Safety Alignment via Reward Gap Optimizationcited by 18
  3. 2025On the Robustness of Reward Models for Language Model Alignmentcited by 13
  4. 2025Pushing on Multilingual Reasoning Models with Language-Mixed Chain-of-Thoughtcited by 10
  5. 2025Implicit Cross-Lingual Rewarding for Efficient Multilingual Preference Alignmentcited by 9
  6. 2025Demystifying Multilingual Chain-of-Thought in Process Reward Modelingcited by 6
  7. 2025Relic: Enhancing Reward Model Generalization for Low-Resource Indic Languages with Few-Shot Examplescited by 6
  8. 2025Making, not Taking, the Best of Ncited by 6
  9. 2025Can RLHF be More Efficient with Imperfect Reward Models? A Policy Coverage Perspectivecited by 4
  10. 2025MENLO: From Preferences to Proficiency -- Evaluating and Modeling Native-like Quality Across 47 Languagescited by 3
  11. 2026A Survey on Rubric-Guided Reinforcement Learning for Language Modelscited by 3
  12. 2026Layer-wise Swapping for Generalizable Multilingual Safetycited by 2
5 more
  1. 2025Cross-Lingual Stability and Bias in Instruction-Tuned Language Models for Humanitarian NLPcited by 1
  2. 2026The Illusion of Cross-Lingual Safety in Low-Resource Languagescited by 1
  3. 2026Neither Here Nor There: Cross-Lingual Representation Dynamics of Code-Mixed Text in Multilingual Encoderscited by 0
  4. 2026CroCo: Cross-Lingual Contrastive Preference Tuning on Self-Generationscited by 0
  5. 2026Same violence, different answer: how AI responds to coercive control against women across languagescited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024M-RewardBench: Evaluating Reward Models in Multilingual Settings
  2. 2025A Survey on Progress in LLM Alignment from the Perspective of Reward Design
  3. 2024Rethinking the Role of Proxy Rewards in Language Model Alignment
  4. 2025Reusing Embeddings: Reproducible Reward Model Research in Large Language Model Alignment without GPUs
  5. 2024RM-Bench: Benchmarking Reward Models of Language Models with Subtlety and Style
  6. 2024Sequence to Sequence Reward Modeling: Improving RLHF by Language Feedback
  7. 2025Beyond Monolithic Rewards: A Hybrid and Multi-Aspect Reward Optimization for MLLM Alignment
  8. 2024West-of-N: Synthetic Preferences for Self-Improving Reward Models