mlsanityData to 14 Oct 2026

EvoPref: Multi-Objective Evolutionary Optimization Discovers Diverse LLM Alignments Beyond Gradient Descent

arXiv 2605.09777 ↗First posted 10 May 2026cs.NE cs.AI cs.CLcited by 0

On the map: LLM alignment / RLHF / preference optimisation → LLM preference optimization methods

Abstract (arXiv)

Gradient-based preference optimization methods for large language model (LLM) alignment suffer from preference collapse, converging to narrow behavioral modes while neglecting preference diversity. We introduce EvoPref, a multi-objective evolutionary algorithm that maintains populations of Low-Rank Adaptation (LoRA) adapters optimized across helpfulness, harmlessness, and honesty objectives using Non-dominated Sorting Genetic Algorithm II (NSGA-II) selection with archive-based diversity preservation. Our primary contribution is demonstrating that population-based methods discover substantially more diverse alignments than gradient descent. On standard benchmarks, EvoPref improves preference coverage by 18% (median 82.5% vs. 70.0% for ORPO, p<0.001, Wilcoxon, n=30) and reduces collapse rates by 47% (11.0% vs. 20.6%, p<0.001), while achieving competitive alignment quality (median 75.5% RewardBench vs. 75.0% for ORPO, p<0.05). We provide theoretical motivation extending recent multi-objective evolutionary algorithm (MOEA) runtime analysis (Dang et al., 2025) suggesting why archive-based methods escape collapse more effectively than single-trajectory optimization. Comprehensive comparisons against MOEA/D, SMS-EMOA, CMA-ES, and gradient baselines (DPO, IPO, KTO, ORPO) with rigorous statistical testing (Friedman with Holm correction, Vargha-Delaney effect sizes, median with IQR) confirm that multi-objective selection with diversity preservation is essential. This work establishes evolutionary optimization as a principled paradigm for diverse LLM alignment.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in LLM preference optimization methods, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 40 of its 67 references that are in the map. First posted on arXiv 10 May 2026.

Builds on

40 of 67 references.

Closest ancestors · 4

Same direction, newest first

  1. 2024ORPO: Monolithic Preference Optimization without Reference Model12 Mar
  2. 2023A General Theoretical Paradigm to Understand Learning from Human Preferences18 Oct
  3. 2022Training language models to follow instructions with human feedback4 Mar
  4. 2021A General Language Assistant as a Laboratory for Alignment1 Dec

Other references · 36

  1. 2017Proximal Policy Optimization Algorithmscited by 15,215
  2. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
  3. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 8,065
  4. 2019Parameter-Efficient Transfer Learning for NLPcited by 4,236
  5. 2021Prefix-Tuning: Optimizing Continuous Prompts for Generationcited by 3,915
  6. 2017Deep reinforcement learning from human preferencescited by 3,912
  7. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,720
  8. 2022Constitutional AI: Harmlessness from AI Feedbackcited by 3,050
  9. 2021TruthfulQA: Measuring How Models Mimic Human Falsehoodscited by 2,975
  10. 2019The Curious Case of Neural Text Degenerationcited by 2,916
  11. 2020Learning to summarize from human feedbackcited by 2,588
  12. 2022Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference timecited by 1,396
24 more
  1. 2023Llama Guard: LLM-based Input-Output Safeguard for Human-AI Conversationscited by 1,207
  2. 2022Editing Models with Task Arithmeticcited by 1,120
  3. 2018Regularized Evolution for Image Classifier Architecture Searchcited by 909
  4. 2018Multi-Task Learning as Multi-Objective Optimizationcited by 821
  5. 2023TIES-Merging: Resolving Interference When Merging Modelscited by 818
  6. 2018Recurrent World Models Facilitate Policy Evolutioncited by 785
  7. 2023Language Models are Super Mario: Absorbing Abilities from Homologous Models as a Free Lunchcited by 694
  8. 2018Loss Surfaces, Mode Connectivity, and Fast Ensembling of DNNscited by 588
  9. 2017Evolution Strategies as a Scalable Alternative to Reinforcement Learningcited by 574
  10. 2018Texygen: A Benchmarking Platform for Text Generation Modelscited by 491
  11. 2015Illuminating search spaces by mapping elitescited by 425
  12. 2023Understanding the Effects of RLHF on LLM Generalisation and Diversitycited by 396
  13. 2016The CMA Evolution Strategy: A Tutorialcited by 377
  14. 2017Population Based Training of Neural Networkscited by 367
  15. 2017Large-Scale Evolution of Image Classifierscited by 348
  16. 2023Fine-Tuning Language Models with Just Forward Passescited by 345
  17. 2023Rewarded soups: towards Pareto-optimal alignment by interpolating weights fine-tuned on diverse rewardscited by 243
  18. 2024Evolutionary Optimization of Model Merging Recipescited by 208
  19. 2024Rainbow Teaming: Open-Ended Generation of Diverse Adversarial Promptscited by 205
  20. 2019Weight Agnostic Neural Networkscited by 71
  21. 2020Illuminating Mario Scenes in the Latent Space of a Generative Adversarial Networkcited by 52
  22. 2022Approximating Gradients for Differentiable Quality Diversity in Reinforcement Learningcited by 38
  23. 2017Safe Mutations for Deep and Recurrent Neural Networks through Output Gradientscited by 15
  24. 2024Rethinking LLM Language Adaptation: A Case Study on Chinese Mixtralcited by 6

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025AMoPO: Adaptive Multi-objective Preference Optimization without Reward Models and Reference Models
  2. 2025ESSA: Evolutionary Strategies for Scalable Alignment
  3. 2025Preference Orchestrator: Prompt-Aware Multi-Objective Alignment for Large Language Models
  4. 2025SimPER: A Minimalist Approach to Preference Alignment without Hyperparameters
  5. 2024Panacea: Pareto Alignment via Preference Adaptation for LLMs
  6. 2026Meta-Aligner: Bidirectional Preference-Policy Optimization for Multi-Objective LLMs Alignment
  7. 2025Reward-aware Preference Optimization: A Unified Mathematical Framework for Model Alignment
  8. 2024Multi-Objective Alignment of Large Language Models Through Hypervolume Maximization