mlsanityData to 7 Oct 2026

Less Data, Better Alignment: Data-Centric Multi-Evaluator Agreement for Preference Optimization

arXiv 2607.25136 ↗First posted 27 Jul 2026cs.AIcited by 0

On the map: LLM alignment / RLHF / preference optimisation → LLM preference optimization methods

Abstract (arXiv)

Research on preference optimization often varies the training objective while holding the data fixed. We instead ask whether a small, high-confidence set of on-policy responses can provide a reliable learning signal. Our method, DMAPO (Data-centric Multi-evaluator Agreement for Preference Optimization), generates candidate responses from the target policy, evaluates helpfulness, factuality, and conciseness with rubric-specialized evaluators, applies a process-critic correction, and retains only high-consensus desirable or undesirable examples. This procedure accepts 1,871 of 54,236 Mistral-7B candidates (3.45%). KTO trained on this set reaches 7.50 on MT-Bench, 95.5% length-controlled win rate against a text-davinci-003 reference, and 57.3% IFEval prompt accuracy. Independent pairwise evaluation also favors DMAPO over SimPO: GPT-4o yields a net win rate of 23.3 points on 129 held-out prompts and 24.0 points on 200 out-of-distribution LMSYS-Chat prompts; Claude Opus 4.7 yields 24.1 points on the held-out set. Changing the evaluator model or rubric alters the selected examples but has little effect on downstream performance. A second-backbone study yields a similar 3.41% acceptance rate, although its performance gains are more modest. Across these experiments, consensus filtering offers a data-efficient route to preference optimization for general instructions, at the cost of additional curation compute and dependence on evaluator judgments.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in LLM preference optimization methods, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 23 of its 27 references that are in the map. First posted on arXiv 27 Jul 2026.

Builds on

23 of 27 references.

Closest ancestors · 5

Same direction, newest first

  1. 2024SimPO: Simple Preference Optimization with a Reference-Free Reward23 May
  2. 2024ORPO: Monolithic Preference Optimization without Reference Model12 Mar
  3. 2023A General Theoretical Paradigm to Understand Learning from Human Preferences18 Oct
  4. 2022Self-Instruct: Aligning Language Models with Self-Generated Instructions20 Dec
  5. 2022Training language models to follow instructions with human feedback4 Mar

Other references · 18

  1. 2017Proximal Policy Optimization Algorithmscited by 14,441
  2. 2024The Llama 3 Herd of Modelscited by 14,192
  3. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  4. 2025Qwen3 Technical Reportcited by 8,195
  5. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,916
  6. 2017Deep reinforcement learning from human preferencescited by 3,846
  7. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,634
  8. 2020Learning to summarize from human feedbackcited by 2,562
  9. 2024Mixtral of Expertscited by 1,785
  10. 2023Instruction-Following Evaluation for Large Language Modelscited by 1,143
  11. 2024Length-Controlled AlpacaEval: A Simple Way to Debias Automatic Evaluatorscited by 865
  12. 2023RLAIF vs. RLHF: Scaling Reinforcement Learning from Human Feedback with AI Feedbackcited by 615
6 more
  1. 2024LESS: Selecting Influential Data for Targeted Instruction Tuningcited by 514
  2. 2025LIMO: Less is More for Reasoningcited by 488
  3. 2023UltraFeedback: Boosting Language Models with Scaled AI Feedbackcited by 394
  4. 2025REINFORCE++: Stabilizing Critic-Free Policy Optimization with Global Advantage Normalizationcited by 233
  5. 2024HelpSteer2: Open-source dataset for training top-performing reward modelscited by 232
  6. 2024Self-Play Preference Optimization for Language Model Alignmentcited by 228

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025MaPPO: Maximum a Posteriori Preference Optimization with Prior Knowledge
  2. 2025InCo-DPO: Balancing Distribution Shift and Data Quality for Enhanced Preference Optimization
  3. 2025Finding the Sweet Spot: Preference Data Construction for Scaling Preference Optimization
  4. 2026AdaDPO: Self-Adaptive Direct Preference Optimization with Balanced Gradient Updates
  5. 2025Understanding the Impact of Sampling Quality in Direct Preference Optimization
  6. 2025When Data is the Algorithm: A Systematic Study and Curation of Preference Optimization Datasets
  7. 2025Uni-DPO: A Unified Paradigm for Dynamic Preference Optimization of LLMs
  8. 2024A Comprehensive Survey of Direct Preference Optimization: Datasets, Theories, Variants, and Applications