mlsanityData to 14 Oct 2026

CAPO: Confidence Aware Preference Optimization Learning for Multilingual Preferences

arXiv 2511.07691 ↗First posted 10 Nov 2025cs.CL cs.AIcited by 3

On the map: LLM alignment / RLHF / preference optimisation → LLM preference optimization methods

Abstract (arXiv)

Preference optimization is a critical post-training technique used to align large language models (LLMs) with human preferences, typically by fine-tuning on ranked response pairs. While methods like Direct Preference Optimization (DPO) have proven effective in English, they often fail to generalize robustly to multilingual settings. We propose a simple yet effective alternative, Confidence-Aware Preference Optimization (CAPO), which replaces DPO's fixed treatment of preference pairs with a dynamic loss scaling mechanism based on a relative reward. By modulating the learning signal according to the confidence in each preference pair, CAPO enhances robustness to noisy or low-margin comparisons, typically encountered in multilingual text. Empirically, CAPO outperforms existing preference optimization baselines by at least 16% in reward accuracy, and improves alignment by widening the gap between preferred and dispreferred responses across languages.

Key facts. Cited by 3 papers in the map (data through 14 Oct 2026). Placed in LLM preference optimization methods, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 25 of its 35 references that are in the map. First posted on arXiv 10 Nov 2025.

Builds on

25 of 35 references.

Closest ancestors · 9

Same direction, newest first

  1. 2024Cal-DPO: Calibrated Direct Preference Optimization for Language Model Alignment19 Dec
  2. 2024RLHF Can Speak Many Languages: Unlocking Multilingual Preference Optimization for LLMs2 Jul
  3. 2024A Deep Dive into the Trade-Offs of Parameter-Efficient Preference Alignment Techniques7 Jun
  4. 2024SimPO: Simple Preference Optimization with a Reference-Free Reward23 May
  5. 2024Controllable Preference Optimization: Toward Controllable Multi-Objective Alignment29 Feb
  6. 2024Contrastive Preference Optimization: Pushing the Boundaries of LLM Performance in Machine Translation16 Jan
  7. 2024MAPO: Advancing Multilingual Reasoning through Multilingual Alignment-as-Preference Optimization12 Jan
  8. 2023Zephyr: Direct Distillation of LM Alignment25 Oct
  9. 2023Okapi: Instruction-tuned Large Language Models in Multiple Languages with Reinforcement Learning from Human Feedback29 Jul

Other references · 16

  1. 2017Proximal Policy Optimization Algorithmscited by 15,215
  2. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
  3. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 8,065
  4. 2017Deep reinforcement learning from human preferencescited by 3,912
  5. 2023Large Language Models are not Fair Evaluatorscited by 1,085
  6. 2020SummEval: Re-evaluating Summarization Evaluationcited by 674
  7. 2021XL-Sum: Large-Scale Multilingual Abstractive Summarization for 44 Languagescited by 347
  8. 2021A Statistical Analysis of Summarization Evaluation Metrics using Resampling Methodscited by 72
  9. 2024The Multilingual Alignment Prism: Aligning Global and Local Preferences to Reduce Harmcited by 70
  10. 2022On the Blind Spots of Model-Based Evaluation Metrics for Text Generationcited by 49
  11. 2020MLQE-PE: A Multilingual Quality Estimation and Post-Editing Datasetcited by 47
  12. 2024X-ALMA: Plug & Play Modules and Adaptive Rejection for Quality Translation at Scalecited by 36
4 more
  1. 2024Language Imbalance Driven Rewarding for Multilingual Self-improvingcited by 28
  2. 2024Reuse Your Rewards: Reward Model Transfer for Zero-Shot Cross-Lingual Alignmentcited by 26
  3. 2025M-IFEval: Multilingual Instruction-Following Evaluationcited by 17
  4. 2022DivEMT: Neural Machine Translation Post-Editing Effort Across Typologically Diverse Languagescited by 11

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 1; 2026: 2.

3 papers.

  1. 2025MTQ-Eval: Multilingual Text Quality Evaluation for Language Modelscited by 0
  2. 2026CroCo: Cross-Lingual Contrastive Preference Tuning on Self-Generationscited by 0
  3. 2026Choosing the Right Language Mode at Inference Time for Multilingual Reliabilitycited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024TSO: Self-Training with Scaled Preference Optimization
  2. 2024Multi-Preference Optimization: Generalizing DPO via Set-Level Contrasts
  3. 2026Language Chain in Alignment: Cross-lingual Ranking Preference Optimization
  4. 2024Self-Augmented Preference Optimization: Off-Policy Paradigms for Language Model Alignment
  5. 2024Towards Improved Preference Optimization Pipeline: from Data Generation to Budget-Controlled Regularization
  6. 2025Implicit Cross-Lingual Rewarding for Efficient Multilingual Preference Alignment
  7. 2025AMoPO: Adaptive Multi-objective Preference Optimization without Reward Models and Reference Models
  8. 2024Towards a Unified View of Preference Learning for Large Language Models: A Survey