mlsanityData to 7 Oct 2026

CroCo: Cross-Lingual Contrastive Preference Tuning on Self-Generations

arXiv 2605.26293 ↗First posted 25 May 2026cs.CL cs.AIcited by 0

On the map: LLM alignment / RLHF / preference optimisation → LLM preference optimization methods

Abstract (arXiv)

Prior work establishes that controlled contrastiveness between self-generated responses from large language models, set via reward scores, improves downstream preference tuning in English. We extend this method to multiple languages and evaluate two models across a total of 14 high and low-resource languages on a diverse set of tasks. Our central finding is that cross-lingual contrastive preference tuning on self-generations (CroCo) transfers without language-specific preference annotation. A reward model trained on English preferences (atop a multilingual base) produces useful within-language rankings across most languages, and pairing in either a monolingual or multilingual setting improves over each model on the majority of setups while preventing the catastrophic forgetting of supervised fine-tuning. We observe that the gains require on-policy data. Off-policy responses reduce the benefit and online preference optimization fails to improve over the offline variant. Specifically, on structured tasks, our method matches or exceeds the base in 6/7 languages for EuroLLM-9B and 4/7 settings for Aya-3B. On open-ended generation, evaluated by two judges, both tuned models win 28/30 times against their respective base across 15 evaluated languages (high and low-resource). Overall, we show promising directions for multilingual preference tuning using self-generations.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in LLM preference optimization methods, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 38 of its 55 references that are in the map. First posted on arXiv 25 May 2026.

Builds on

38 of 55 references.

Closest ancestors · 6

Same direction, newest first

  1. 2025CAPO: Confidence Aware Preference Optimization Learning for Multilingual Preferences10 Nov
  2. 2025What Matters in Data for DPO?23 Aug
  3. 2025Finding the Sweet Spot: Preference Data Construction for Scaling Preference Optimization24 Feb
  4. 2024RLHF Can Speak Many Languages: Unlocking Multilingual Preference Optimization for LLMs2 Jul
  5. 2024Direct Language Model Alignment from Online AI Feedback7 Feb
  6. 2024MAPO: Advancing Multilingual Reasoning through Multilingual Alignment-as-Preference Optimization12 Jan

Other references · 32

  1. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,916
  2. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,767
  3. 2016SQuAD: 100,000+ Questions for Machine Comprehension of Textcited by 4,115
  4. 2024MMLU-Pro: A More Robust and Challenging Multi-Task Language Understanding Benchmarkcited by 1,758
  5. 2024Length-Controlled AlpacaEval: A Simple Way to Debias Automatic Evaluatorscited by 865
  6. 2023An Empirical Study of Catastrophic Forgetting in Large Language Models During Continual Fine-tuningcited by 622
  7. 2019Quantifying the Carbon Emissions of Machine Learningcited by 516
  8. 2019MLQA: Evaluating Cross-lingual Extractive Question Answeringcited by 339
  9. 2024Skywork-Reward: Bag of Tricks for Reward Modeling in LLMscited by 322
  10. 2024LLMs instead of Human Judges? A Large Scale Empirical Study across 20 NLP Evaluation Taskscited by 236
  11. 2024Continual Learning of Large Language Models: A Comprehensive Surveycited by 228
  12. 2025Skywork-Reward-V2: Scaling Preference Data Curation via Human-AI Synergycited by 204
20 more
  1. 2024Preference Fine-Tuning of LLMs Should Leverage Suboptimal, On-Policy Datacited by 184
  2. 2025RewardBench 2: Advancing Reward Model Evaluationcited by 136
  3. 2024INCLUDE: Evaluating Multilingual Language Understanding with Regional Knowledgecited by 78
  4. 2021Machine Translationese: Effects of Algorithmic Bias on Linguistic Complexity in Machine Translationcited by 70
  5. 2020FQuAD: French Question Answering Datasetcited by 58
  6. 2024M-RewardBench: Evaluating Reward Models in Multilingual Settingscited by 50
  7. 2021GermanQuAD and GermanDPR: Improving Non-English Question Answering and Passage Retrievalcited by 39
  8. 2024Encoder vs Decoder: Comparative Analysis of Encoder and Decoder Language Models on Multilingual NLU Taskscited by 33
  9. 2024Language Imbalance Driven Rewarding for Multilingual Self-improvingcited by 27
  10. 2024Reuse Your Rewards: Reward Model Transfer for Zero-Shot Cross-Lingual Alignmentcited by 26
  11. 2025The Delta Learning Hypothesis: Preference Tuning on Weak Data can Yield Strong Gainscited by 25
  12. 2025MultiLoKo: a multilingual local knowledge benchmark for LLMs spanning 31 languagescited by 22
  13. 2024Cross-lingual Transfer of Reward Models in Multilingual Alignmentcited by 17
  14. 2024Fine-Tuning Large Language Models to Translate: Will a Touch of Noisy Data in Misaligned Languages Suffice?cited by 16
  15. 2025MultiNRC: A Challenging and Native Multilingual Reasoning Evaluation Benchmark for LLMscited by 16
  16. 2026Crosslingual On-Policy Self-Distillation for Multilingual Reasoningcited by 10
  17. 2026SFT-then-RL Outperforms Mixed-Policy Methods for LLM Reasoningcited by 8
  18. 2025MultiWikiQA: A Reading Comprehension Benchmark in 300+ Languagescited by 7
  19. 2023DUMB: A Benchmark for Smart Evaluation of Dutch Modelscited by 6
  20. 2025DaLA: Danish Linguistic Acceptability Evaluation Guided by Real World Errorscited by 3

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026Language Chain in Alignment: Cross-lingual Ranking Preference Optimization
  2. 2025Implicit Cross-Lingual Rewarding for Efficient Multilingual Preference Alignment
  3. 2024Self-Augmented Preference Optimization: Off-Policy Paradigms for Language Model Alignment
  4. 2026Macro: Enhancing Multilingual Counterfactual Explanations through Alignment-as-Preference Optimization
  5. 2025Temporal Self-Rewarding Language Models: Decoupling Chosen-Rejected via Past-Future
  6. 2024Aligning Large Language Models by On-Policy Self-Judgment
  7. 2026Meta-Learning Preferences for Multilingual LLM Alignment
  8. 2024Improving Context-Aware Preference Modeling for Language Models