mlsanityData to 14 Oct 2026

CONGRAD:Conflicting Gradient Filtering for Multilingual Preference Alignment

arXiv 2503.23777 ↗First posted 31 Mar 2025cs.CLcited by 3

On the map: LLM alignment / RLHF / preference optimisation → LLM preference optimization methods

Abstract (arXiv)

Naive joint training of large language models (LLMs) for multilingual preference alignment can suffer from negative interference. This is a known issue in multilingual training, where conflicting objectives degrade overall performance. However, the impact of this phenomenon in the context of multilingual preference alignment remains largely underexplored. To address this issue, we propose CONGRAD, a scalable and effective filtering method that selects high-quality preference samples with minimal gradient conflicts across languages. Our method leverages gradient surgery to retain samples aligned with an aggregated multilingual update direction. Additionally, we incorporate a sublinear gradient compression strategy that reduces memory overhead during gradient accumulation. We integrate CONGRAD into self-rewarding framework and evaluate on LLaMA3-8B and Gemma2-2B across 10 languages. Results show that CONGRAD consistently outperforms strong baselines in both seen and unseen languages, with minimal alignment tax.

Key facts. Cited by 3 papers in the map (data through 14 Oct 2026). Placed in LLM preference optimization methods, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 50 of its 66 references that are in the map. First posted on arXiv 31 Mar 2025.

Builds on

50 of 66 references.

Closest ancestors · 8

Same direction, newest first

  1. 2025Implicit Cross-Lingual Rewarding for Efficient Multilingual Preference Alignment6 Mar
  2. 2024RLHF Can Speak Many Languages: Unlocking Multilingual Preference Optimization for LLMs2 Jul
  3. 2024Iterative Length-Regularized Direct Preference Optimization: A Case Study on Improving 7B Language Models to GPT-4 Level17 Jun
  4. 2024Filtered Direct Preference Optimization22 Apr
  5. 2023Mitigating the Alignment Tax of RLHF12 Sep
  6. 2023Okapi: Instruction-tuned Large Language Models in Multiple Languages with Reinforcement Learning from Human Feedback29 Jul
  7. 2023LIMA: Less Is More for Alignment18 May
  8. 2022Training language models to follow instructions with human feedback4 Mar

Other references · 42

  1. 2017Proximal Policy Optimization Algorithmscited by 15,215
  2. 2024The Llama 3 Herd of Modelscited by 14,715
  3. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  4. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 8,065
  5. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,944
  6. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,886
  7. 2024GPT-4o System Cardcited by 4,588
  8. 2019Unsupervised Cross-lingual Representation Learning at Scalecited by 3,935
  9. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,720
  10. 2019Fine-Tuning Language Models from Human Preferencescited by 2,150
  11. 2024Gemma 2: Improving Open Language Models at a Practical Sizecited by 1,998
  12. 2020Gradient Surgery for Multi-Task Learningcited by 1,256
30 more
  1. 2018Multi-Task Learning as Multi-Objective Optimizationcited by 821
  2. 2023AlpacaFarm: A Simulation Framework for Methods that Learn from Human Feedbackcited by 748
  3. 2024Self-Rewarding Language Modelscited by 681
  4. 2023RLAIF vs. RLHF: Scaling Reinforcement Learning from Human Feedback with AI Feedbackcited by 625
  5. 2024Self-Play Fine-Tuning Converts Weak Language Models to Strong Language Modelscited by 557
  6. 2024LESS: Selecting Influential Data for Targeted Instruction Tuningcited by 532
  7. 2023UltraFeedback: Boosting Language Models with Scaled AI Feedbackcited by 406
  8. 2023Understanding the Effects of RLHF on LLM Generalisation and Diversitycited by 396
  9. 2019Massively Multilingual Neural Machine Translation in the Wild: Findings and Challengescited by 257
  10. 2024Iterative Reasoning Preference Optimizationcited by 246
  11. 2023Multilingual Jailbreak Challenges in Large Language Modelscited by 240
  12. 2024Global MMLU: Understanding and Addressing Cultural and Linguistic Biases in Multilingual Evaluationcited by 191
  13. 2023Gender bias and stereotypes in Large Language Modelscited by 187
  14. 2024Aya Dataset: An Open-Access Collection for Multilingual Instruction Tuningcited by 177
  15. 2020Gradient Vaccine: Investigating and Improving Multi-task Optimization in Massively Multilingual Modelscited by 169
  16. 2024Flora: Low-Rank Adapters Are Secretly Gradient Compressorscited by 108
  17. 2024DART-Math: Difficulty-Aware Rejection Tuning for Mathematical Problem-Solvingcited by 95
  18. 2024Multilingual Instruction Tuning With Just a Pinch of Multilingualitycited by 73
  19. 2023Monolingual or Multilingual Instruction Tuning: Which Makes a Better Alpacacited by 60
  20. 2024M-RewardBench: Evaluating Reward Models in Multilingual Settingscited by 50
  21. 2023GPTAraEval: A Comprehensive Evaluation of ChatGPT on Arabic NLPcited by 37
  22. 2024Enhancing Multilingual Capabilities of Large Language Models through Self-Distillation from Resource-Rich Languagescited by 36
  23. 2023Improving In-context Learning of Multilingual Generative Language Models with Cross-lingual Alignmentcited by 33
  24. 2023On Evaluating and Mitigating Gender Biases in Multilingual Settingscited by 31
  25. 2024Language Imbalance Driven Rewarding for Multilingual Self-improvingcited by 28
  26. 2025Beyond English: The Impact of Prompt Translation Strategies across Languages and Tasks in Multilingual LLMscited by 22
  27. 2021Uncertainty-Aware Balancing for Multilingual and Multi-Domain Neural Machine Translation Trainingcited by 19
  28. 2024Mixture-of-Skills: Learning to Optimize Data Usage for Fine-Tuning Large Language Modelscited by 16
  29. 2017A Multifaceted Evaluation of Neural versus Phrase-Based Machine Translation for 9 Language Directionscited by 10
  30. 2023Order Matters in the Presence of Dataset Imbalance for Multilingual Learningcited by 7

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 3.

3 papers.

  1. 2026COMPASS: COntinual Multilingual PEFT with Adaptive Semantic Samplingcited by 0
  2. 2026Multilingual Fine-Tuning via Localized Gradient Conflict Resolutioncited by 0
  3. 2026Meta-Learning Preferences for Multilingual LLM Alignmentcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025CM-Align: Consistency-based Multilingual Alignment for Large Language Models
  2. 2026Language Chain in Alignment: Cross-lingual Ranking Preference Optimization
  3. 2025SimPER: A Minimalist Approach to Preference Alignment without Hyperparameters
  4. 2024A Deep Dive into the Trade-Offs of Parameter-Efficient Preference Alignment Techniques
  5. 2024Towards a Unified View of Preference Learning for Large Language Models: A Survey
  6. 2026Meta-Aligner: Bidirectional Preference-Policy Optimization for Multi-Objective LLMs Alignment
  7. 2025Preference Orchestrator: Prompt-Aware Multi-Objective Alignment for Large Language Models
  8. 2025Alignment through Meta-Weighted Online Sampling: Bridging the Gap between Data Generation and Preference Optimization