mlsanityData to 14 Oct 2026

Multi-objective Large Language Model Alignment with Hierarchical Experts

arXiv 2505.20925 ↗First posted 27 May 2025cs.CL cs.AIcited by 13

On the map: LLM alignment / RLHF / preference optimisation → LLM preference optimization methods

Abstract (arXiv)

Aligning large language models (LLMs) to simultaneously satisfy multiple objectives remains a significant challenge, especially given the diverse and often conflicting nature of human preferences. Existing alignment methods struggle to balance trade-offs effectively, often requiring costly retraining or yielding suboptimal results across the Pareto frontier of preferences. In this paper, we introduce textitHoE(Hierarchical Mixture-of-Experts), a textitlightweight, textitparameter-efficient, and textitplug-and-play approach that eliminates the need for model training, while enabling LLMs to adapt across the entire Pareto frontier and accommodate diverse user preferences. In particular, textitHoE consists of three hierarchical components: LoRA Experts, Router Experts and Preference Routing, reaching optimal Pareto frontiers and achieving a trade-off between parameter size, training cost, and performance. We evaluate textitHoE across various tasks on 14 objectives and 200 different preferences among 6 benchmarks, demonstrating superior performance over 15 recent baselines. Code is available in the supplementary materials.

Key facts. Cited by 13 papers in the map (data through 14 Oct 2026). Placed in LLM preference optimization methods, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 56 of its 70 references that are in the map. First posted on arXiv 27 May 2025.

Builds on

56 of 70 references.

Closest ancestors · 10

Same direction, newest first

  1. 2025PARM: Multi-Objective Test-Time Alignment via Preference-Aware Autoregressive Reward Model6 May
  2. 2024Multi-Objective Alignment of Large Language Models Through Hypervolume Maximization6 Dec
  3. 2024GenARM: Reward Guided Generation with Autoregressive Reward Model for Test-time Alignment10 Oct
  4. 2024Decoding-Time Language Model Alignment with Multiple Objectives27 Jun
  5. 2024MetaAligner: Towards Generalizable Multi-Objective Alignment of Language Models25 Mar
  6. 2024Arithmetic Control of LLMs for Diverse User Preferences: Directional Preference Alignment with Multi-Objective Rewards28 Feb
  7. 2024Rewards-in-Context: Multi-objective Alignment of Foundation Models with Dynamic Preference Adjustment15 Feb
  8. 2024Aligner: Efficient Alignment by Learning to Correct4 Feb
  9. 2024ARGS: Alignment as Reward-Guided Search23 Jan
  10. 2023Beyond One-Preference-Fits-All Alignment: Multi-Objective Direct Preference Optimization5 Oct

Other references · 46

  1. 2023GPT-4 Technical Reportcited by 18,396
  2. 2017Proximal Policy Optimization Algorithmscited by 15,215
  3. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  4. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  5. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,720
  6. 2020Learning to summarize from human feedbackcited by 2,588
  7. 2019Fine-Tuning Language Models from Human Preferencescited by 2,150
  8. 2022Editing Models with Task Arithmeticcited by 1,120
  9. 2023Steering Llama 2 via Contrastive Activation Additioncited by 1,085
  10. 2023TIES-Merging: Resolving Interference When Merging Modelscited by 818
  11. 2021Merging Models with Fisher-Weighted Averagingcited by 621
  12. 2022Dataless Knowledge Fusion by Merging Weights of Language Modelscited by 387
34 more
  1. 2016The CMA Evolution Strategy: A Tutorialcited by 377
  2. 2023Personalized Soups: Personalized Large Language Model Alignment via Post-hoc Parameter Mergingcited by 280
  3. 2023AdaMerging: Adaptive Model Merging for Multi-Task Learningcited by 252
  4. 2024OR-Bench: An Over-Refusal Benchmark for Large Language Modelscited by 244
  5. 2023Rewarded soups: towards Pareto-optimal alignment by interpolating weights fine-tuned on diverse rewardscited by 243
  6. 2024SVD-LLM: Truncation-aware Singular Value Decomposition for Large Language Model Compressioncited by 243
  7. 2023ASVD: Activation-aware Singular Value Decomposition for Compressing Large Language Modelscited by 169
  8. 2023Pushing Mixture of Experts to the Limit: Extremely Parameter Efficient MoE for Instruction Tuningcited by 150
  9. 2024Task Singular Vectors: Reducing Task Interference in Model Mergingcited by 147
  10. 2024Higher Layers Need More LoRA Expertscited by 102
  11. 2019Deep Reinforcement Learning for Multi-objective Optimizationcited by 90
  12. 2019Safe Policies for Reinforcement Learning via Primal-Dual Methodscited by 79
  13. 2024Parameter Competition Balancing for Model Mergingcited by 72
  14. 2024Style Vectors for Steering Generative Large Language Modelcited by 56
  15. 2024Smooth Tchebycheff Scalarization for Multi-Objective Optimizationcited by 54
  16. 2024Multimodal Reasoning with Multimodal Knowledge Graphcited by 44
  17. 2024Conditional Language Policy: A General Framework for Steerable Multi-Objective Finetuningcited by 43
  18. 2024X-LoRA: Mixture of Low-Rank Adapter Experts, a Flexible Framework for Large Language Models with Applications in Protein Mechanics and Molecular Designcited by 42
  19. 2023Query-Dependent Prompt Evaluation and Optimization with Offline Inverse RLcited by 37
  20. 2023Merging Experts into One: Improving Computational Efficiency of Mixture of Expertscited by 37
  21. 2025Delta Decompression for MoE-based LLMs Compressioncited by 33
  22. 2025MTSA: Multi-turn Safety Alignment for LLMs through Multi-round Red-teamingcited by 32
  23. 2024One-Shot Safety Alignment for Large Language Models via Optimal Dualizationcited by 24
  24. 2023Chain of Thought with Explicit Evidence Reasoning for Few-shot Relation Extractioncited by 19
  25. 2024Delta-CoMe: Training-Free Delta-Compression with Mixed-Precision for Large Language Modelscited by 17
  26. 2024Knowledge Fusion By Evolving Weights of Language Modelscited by 16
  27. 2024Traversing Pareto Optimal Policies: Provably Efficient Multi-Objective Reinforcement Learningcited by 15
  28. 2025Safety Alignment via Constrained Knowledge Unlearningcited by 14
  29. 2023Efficient Storage of Fine-Tuned Models via Low-Rank Approximation of Weight Residualscited by 12
  30. 2025Neural Parameter Search for Slimmer Fine-Tuned Models and Better Transfercited by 11
  31. 2024Knowledge Editing with Dynamic Knowledge Graphs for Multi-Hop Question Answeringcited by 10
  32. 2025Multi-Modality Expansion and Retention for LLMs through Parameter Merging and Decouplingcited by 5
  33. 2024Adaptive Online Mirror Descent for Tchebycheff Scalarization in Multi-Objective Learningcited by 4
  34. 2024Impromptu Cybercrime Euphemism Detectioncited by 3

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 1; 2026: 12.

13 papers.

Sort:

By area:

  1. 2025Learning to Optimize Multi-Objective Alignment Through Dynamic Reward Weightingcited by 23
  2. 2026AI Alignment Breaks at the Edgecited by 4
  3. 2026RVPO: Risk-Sensitive Alignment via Variance Regularizationcited by 3
  4. 2026Uncovering Cross-Objective Interference in Multi-Objective Alignmentcited by 2
  5. 2026Personalizing LLMs with Binary Feedback: A Preference-Corrected Optimization Frameworkcited by 1
  6. 2026Dynamic Model Merging Made Slimcited by 1
  7. 2026Skill Weaving: Efficient LLM Improvement via Modular Skillpackscited by 1
  8. 2026Risk-Conditioned Fine-Tuning of Large Language Modelscited by 1
  9. 2026Many Preferences, Few Policies: Towards Scalable Language Model Personalizationcited by 0
  10. 2026SURF: Steering the Scalarization Weight to Uniformly Traverse the Pareto Frontcited by 0
  11. 2026Evolutionary Soups: Evolving Mixture-of-Experts for Multi-Objective LLM Alignmentcited by 0
  12. 2026Open-Qwen-Music: An Auditable Framework for LLM-Based Music Composition and Diffusion Renderingcited by 0
1 more
  1. 2026Which Preferences to Train On? End-to-End Multi-Objective Alignment with an Adversarial Preference Distributioncited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025ParetoHqD: Fast Offline Multiobjective Alignment of Large Language Models using Pareto High-quality Data
  2. 2025Preference Orchestrator: Prompt-Aware Multi-Objective Alignment for Large Language Models
  3. 2024Hybrid Alignment Training for Large Language Models
  4. 2025MAVIS: Multi-Objective Alignment via Inference-Time Value-Guided Selection
  5. 2025Alignment and Safety in Large Language Models: Safety Mechanisms, Training Paradigms, and Emerging Challenges
  6. 2026Reward-free Alignment for Conflicting Objectives
  7. 2025Pareto Multi-Objective Alignment for Language Models
  8. 2024Panacea: Pareto Alignment via Preference Adaptation for LLMs