mlsanityData to 14 Oct 2026

PMoL: Parameter Efficient MoE for Preference Mixing of LLM Alignment

arXiv 2411.01245 ↗First posted 2 Nov 2024cs.CLcited by 1

On the map: LLM alignment / RLHF / preference optimisation → LLM preference optimization methods

also in 1 more
Abstract (arXiv)

Reinforcement Learning from Human Feedback (RLHF) has been proven to be an effective method for preference alignment of large language models (LLMs) and is widely used in the post-training process of LLMs. However, RLHF struggles with handling multiple competing preferences. This leads to a decrease in the alignment of LLMs with human preferences. To address this issue, we propose Preference Mixture of LoRAs (PMoL) from the perspective of model architecture, which can adapt to any number of preferences to mix. PMoL combines Mixture of Experts (MoE) and Low Rank Adaptor (LoRA). This architecture is innovatively applied to the research of preference alignment and has achieved significant performance improvement. The expert group soft loss is used to enable MoE with the ability to mix preferences. Through comprehensive evaluation by the reward model and GPT-4o, the experiment results show that PMoL has superior preference mixing capabilities compared to baseline methods. PMoL achieves better preference alignment with lower training costs.

Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Placed in LLM preference optimization methods, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 34 of its 41 references that are in the map. First posted on arXiv 2 Nov 2024.

Builds on

34 of 41 references.

Closest ancestors · 4

Same direction, newest first

  1. 2024ORPO: Monolithic Preference Optimization without Reference Model12 Mar
  2. 2024Aligning Crowd Feedback via Distributional Preference Reward Modeling15 Feb
  3. 2023The Unlocking Spell on Base LLMs: Rethinking Alignment via In-Context Learning4 Dec
  4. 2022Training language models to follow instructions with human feedback4 Mar

Other references · 30

  1. 2023GPT-4 Technical Reportcited by 18,396
  2. 2017Proximal Policy Optimization Algorithmscited by 15,215
  3. 2024The Llama 3 Herd of Modelscited by 14,715
  4. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
  5. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 8,065
  6. 2017Deep reinforcement learning from human preferencescited by 3,912
  7. 2021The Power of Scale for Parameter-Efficient Prompt Tuningcited by 3,795
  8. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,720
  9. 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layercited by 3,509
  10. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,073
  11. 2020GShard: Scaling Giant Models with Conditional Computation and Automatic Shardingcited by 1,631
  12. 2021GLaM: Efficient Scaling of Language Models with Mixture-of-Expertscited by 897
18 more
  1. 2022Few-Shot Parameter-Efficient Fine-Tuning is Better and Cheaper than In-Context Learningcited by 896
  2. 2023BeaverTails: Towards Improved Safety Alignment of LLM via a Human-Preference Datasetcited by 860
  3. 2021Scaling Vision with Sparse Mixture of Expertscited by 747
  4. 2023Safe RLHF: Safe Reinforcement Learning from Human Feedbackcited by 656
  5. 2023LoraHub: Efficient Cross-Task Generalization via Dynamic LoRA Compositioncited by 364
  6. 2021Towards Emotional Support Dialog Systemscited by 299
  7. 2023Personalized Soups: Personalized Large Language Model Alignment via Post-hoc Parameter Mergingcited by 280
  8. 2023Prompt Cache: Modular Attention Reuse for Low-Latency Inferencecited by 268
  9. 2023The False Promise of Imitating Proprietary LLMscited by 263
  10. 2023Rewarded soups: towards Pareto-optimal alignment by interpolating weights fine-tuned on diverse rewardscited by 243
  11. 2022Fine-tuning language models to find agreement among humans with diverse preferencescited by 189
  12. 2023S-LoRA: Serving Thousands of Concurrent LoRA Adapterscited by 152
  13. 2023Platypus: Quick, Cheap, and Powerful Refinement of LLMscited by 150
  14. 2023Pushing Mixture of Experts to the Limit: Extremely Parameter Efficient MoE for Instruction Tuningcited by 150
  15. 2024MaxMin-RLHF: Alignment with Diverse Human Preferencescited by 123
  16. 2024Aligning to Thousands of Preferences via System Message Generalizationcited by 84
  17. 2023Human Feedback is not Gold Standardcited by 83
  18. 2024PEMT: Multi-Task Correlation Guided Mixture-of-Experts Enables Parameter-Efficient Transfer Learningcited by 7

Built on by

Cited by year

Citing papers per year: 2025: 1.

1 paper.

  1. 2025LEO-MINI: An Efficient Multimodal Large Language Model using Conditional Token Reduction and Mixture of Multi-Modal Expertscited by 4

Similar, not cited

Close in wording, no citation either way

  1. 2024Mixed Preference Optimization: Reinforcement Learning with Data Selection and Better Reference Model
  2. 2025Beyond RLHF: A Unified Theoretical Framework of Alignment
  3. 2026From RLHF to Direct Alignment: A Theoretical Unification of Preference Learning for Large Language Models
  4. 2024Accelerated Preference Optimization for Large Language Model Alignment
  5. 2025Mix- and MoE-DPO: A Variational Inference Approach to Direct Preference Optimization
  6. 2024Unified Preference Optimization: Language Model Alignment Beyond the Preference Frontier
  7. 2025MPO: An Efficient Post-Processing Framework for Mixing Diverse Preference Alignment
  8. 2025RE-PO: Robust Enhanced Policy Optimization as a General Framework for LLM Alignment