mlsanityData to 14 Oct 2026

Meta-Learning Preferences for Multilingual LLM Alignment

arXiv 2607.13315 ↗First posted 14 Jul 2026cs.CLcited by 0

On the map: LLM alignment / RLHF / preference optimisation → LLM preference optimization methods

Abstract (arXiv)

Unequal availability of human preference data across languages poses a significant challenge for aligning large language models in multilingual settings. To address the lack of sufficient data in low-resource language alignment, we propose a meta-learning framework for Reinforcement Learning from Human Feedback and Direct Preference Optimization. By leveraging preference data from other languages, our framework learns a transferable initialization that enables effective adaptation to a target language with minimal data. We provide theoretical guarantees for both the meta-reward modeling and meta-policy optimization settings, and empirically demonstrate the effectiveness of our approach on multilingual benchmarks. In an extremely low-resource setting with only 100 target-language preference samples, our approach achieves up to 28% win-rate improvements over baseline methods, and consistently outperforms baselines across multiple target languages and model scales. Our approaches retain these advantages across different combinations of meta-training languages and varying linguistic distances from the target languages.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in LLM preference optimization methods, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 40 of its 52 references that are in the map. First posted on arXiv 14 Jul 2026.

Builds on

40 of 52 references.

Closest ancestors · 9

Same direction, newest first

  1. 2025CONGRAD:Conflicting Gradient Filtering for Multilingual Preference Alignment31 Mar
  2. 2024Right Now, Wrong Then: Non-Stationary Direct Preference Optimization under Preference Drift26 Jul
  3. 2024LLMs Beyond English: Scaling the Multilingual Capability of LLMs with Cross-Lingual Feedback3 Jun
  4. 2024SimPO: Simple Preference Optimization with a Reference-Free Reward23 May
  5. 2024From r to Q^*: Your Language Model is Secretly a Q-Function18 Apr
  6. 2024KTO: Model Alignment as Prospect Theoretic Optimization2 Feb
  7. 2023A General Theoretical Paradigm to Understand Learning from Human Preferences18 Oct
  8. 2023Okapi: Instruction-tuned Large Language Models in Multiple Languages with Reinforcement Learning from Human Feedback29 Jul
  9. 2022Training language models to follow instructions with human feedback4 Mar

Other references · 31

  1. 2023GPT-4 Technical Reportcited by 18,396
  2. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  3. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 8,065
  4. 2020Measuring Massive Multitask Language Understandingcited by 7,963
  5. 2017Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networkscited by 4,897
  6. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,886
  7. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,100
  8. 2025Gemini 2.5: Pushing the Frontier with Advanced Reasoning, Multimodality, Long Context, and Next Generation Agentic Capabilitiescited by 3,796
  9. 2023Qwen Technical Reportcited by 3,344
  10. 2016Matching Networks for One Shot Learningcited by 2,255
  11. 2019Fine-Tuning Language Models from Human Preferencescited by 2,150
  12. 2024Gemma 2: Improving Open Language Models at a Practical Sizecited by 1,998
19 more
  1. 2025Gemma 3 Technical Reportcited by 1,824
  2. 2020Meta-Learning in Neural Networks: A Surveycited by 950
  3. 2024Length-Controlled AlpacaEval: A Simple Way to Debias Automatic Evaluatorscited by 897
  4. 2021Meta-learning via Language Model In-context Tuningcited by 164
  5. 2019On the Convergence Theory of Gradient-Based Model-Agnostic Meta-Learning Algorithmscited by 132
  6. 2024LLMs Are Few-Shot In-Context Low-Resource Language Learnerscited by 68
  7. 2024Pangea: A Fully Open Multilingual Multimodal LLM for 39 Languagescited by 64
  8. 2023Investigating the Translation Performance of a Large Multilingual Language Model: the Case of BLOOMcited by 59
  9. 2023Meta-in-context learning in large language modelscited by 59
  10. 2023Meta-Learning Online Adaptation of Language Modelscited by 43
  11. 2023Improving In-context Learning of Multilingual Generative Language Models with Cross-lingual Alignmentcited by 33
  12. 2024Towards Multilingual LLM Evaluation for European Languagescited by 30
  13. 2024Reward Model Learning vs. Direct Policy Optimization: A Comparative Analysis of Learning from Human Preferencescited by 27
  14. 2024MEND: Meta dEmonstratioN Distillation for Efficient and Effective In-Context Learningcited by 15
  15. 2024Meta In-Context Learning Makes Large Language Models Better Zero and Few-Shot Relation Extractorscited by 13
  16. 2024MAML-en-LLM: Model Agnostic Meta-Training of LLMs for Improved In-Context Learningcited by 10
  17. 2025Krutrim LLM: Multilingual Foundational Model for over a Billion Peoplecited by 10
  18. 2024Evaluating and Mitigating Linguistic Discrimination in Large Language Modelscited by 8
  19. 2025LLMs as In-Context Meta-Learners for Model and Hyperparameter Selectioncited by 3

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2024Unified Preference Optimization: Language Model Alignment Beyond the Preference Frontier
  2. 2024MetaAlign: Align Large Language Models with Diverse Preferences during Inference Time
  3. 2026Meta-Aligner: Bidirectional Preference-Policy Optimization for Multi-Objective LLMs Alignment
  4. 2025Implicit Cross-Lingual Rewarding for Efficient Multilingual Preference Alignment
  5. 2024Towards a Unified View of Preference Learning for Large Language Models: A Survey
  6. 2025Alignment through Meta-Weighted Online Sampling: Bridging the Gap between Data Generation and Preference Optimization
  7. 2024Investigating on RLHF methodology
  8. 2025CM-Align: Consistency-based Multilingual Alignment for Large Language Models