mlsanityData to 7 Oct 2026

ParetoHqD: Fast Offline Multiobjective Alignment of Large Language Models using Pareto High-quality Data

arXiv 2504.16628 ↗First posted 23 Apr 2025cs.LG cs.CLcited by 4

On the map: LLM alignment / RLHF / preference optimisation → LLM preference optimization methods

Abstract (arXiv)

Aligning large language models with multiple human expectations and values is crucial for ensuring that they adequately serve a variety of user needs. To this end, offline multiobjective alignment algorithms such as the Rewards-in-Context algorithm have shown strong performance and efficiency. However, inappropriate preference representations and training with imbalanced reward scores limit the performance of such algorithms. In this work, we introduce ParetoHqD that addresses the above issues by representing human preferences as preference directions in the objective space and regarding data near the Pareto front as "high-quality" data. For each preference, ParetoHqD follows a two-stage supervised fine-tuning process, where each stage uses an individual Pareto high-quality training set that best matches its preference direction. The experimental results have demonstrated the superiority of ParetoHqD over five baselines on two multiobjective alignment tasks.

Key facts. Cited by 4 papers in the map (data through 7 Oct 2026). Placed in LLM preference optimization methods, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 30 of its 45 references that are in the map. First posted on arXiv 23 Apr 2025.

Builds on

30 of 45 references.

Closest ancestors · 15

Same direction, newest first

  1. 2024Unlocking Decoding-time Controllability: Gradient-Free Multi-Objective Alignment with Contrastive Prompts9 Aug
  2. 2024β-DPO: Direct Preference Optimization with Dynamic β11 Jul
  3. 2024Decoding-Time Language Model Alignment with Multiple Objectives27 Jun
  4. 2024MetaAligner: Towards Generalizable Multi-Objective Alignment of Language Models25 Mar
  5. 2024Provably Robust DPO: Aligning Language Models with Noisy Feedback1 Mar
  6. 2024Controllable Preference Optimization: Toward Controllable Multi-Objective Alignment29 Feb
  7. 2024Arithmetic Control of LLMs for Diverse User Preferences: Directional Preference Alignment with Multi-Objective Rewards28 Feb
  8. 2024Rewards-in-Context: Multi-objective Alignment of Foundation Models with Dynamic Preference Adjustment15 Feb
  9. 2024Panacea: Pareto Alignment via Preference Adaptation for LLMs3 Feb
  10. 2023What Makes Good Data for Alignment? A Comprehensive Study of Automatic Data Selection in Instruction Tuning25 Dec
  11. 2023Beyond One-Preference-Fits-All Alignment: Multi-Objective Direct Preference Optimization5 Oct
  12. 2023Statistical Rejection Sampling Improves Preference Optimization13 Sep
3 more
  1. 2023LIMA: Less Is More for Alignment18 May
  2. 2023Fundamental Limitations of Alignment in Large Language Models19 Apr
  3. 2022Training language models to follow instructions with human feedback4 Mar

Other references · 15

  1. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  2. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,916
  3. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,634
  4. 2019The Curious Case of Neural Text Degenerationcited by 2,881
  5. 2020Learning to summarize from human feedbackcited by 2,562
  6. 2023A Survey of Large Language Modelscited by 2,537
  7. 2022Red Teaming Language Models with Language Modelscited by 983
  8. 2023Trustworthy LLMs: a Survey and Guideline for Evaluating Large Language Models' Alignmentcited by 394
  9. 2023Rewarded soups: towards Pareto-optimal alignment by interpolating weights fine-tuned on diverse rewardscited by 232
  10. 2021Delving into Deep Imbalanced Regressioncited by 157
  11. 2023SteerLM: Attribute Conditioned SFT as an (User-Steerable) Alternative to RLHFcited by 107
  12. 2019Deep Reinforcement Learning for Multi-objective Optimizationcited by 87
3 more
  1. 2024Smaller Language Models are capable of selecting Instruction-Tuning Training Data for Larger Language Modelscited by 32
  2. 2024An Experimental Design Framework for Label-Efficient Supervised Finetuning of Large Language Modelscited by 22
  3. 2024Reward Generalization in RLHF: A Topological Perspectivecited by 9

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 2; 2026: 2.

4 papers.

  1. 2025Multi-dimensional Preference Alignment by Conditioning Reward Itselfcited by 1
  2. 2025Solver-Independent Automated Problem Formulation via LLMs for High-Cost Simulation-Driven Designcited by 1
  3. 2026Overlooked Safety Vulnerability in LLMs: Malicious Intelligent Optimization Algorithm Request and its Jailbreakcited by 1
  4. 2026Explaining and Breaking the Safety-Helpfulness Ceiling via Preference Dimensional Expansioncited by 1

Similar, not cited

Close in wording, no citation either way

  1. 2025Multi-objective Large Language Model Alignment with Hierarchical Experts
  2. 2025Pareto Multi-Objective Alignment for Language Models
  3. 2026Reward-free Alignment for Conflicting Objectives
  4. 2025Preference Orchestrator: Prompt-Aware Multi-Objective Alignment for Large Language Models
  5. 2024Multi-Objective Alignment of Large Language Models Through Hypervolume Maximization
  6. 2025Simultaneous Multi-objective Alignment Across Verifiable and Non-verifiable Rewards
  7. 2025UC-MOA: Utility-Conditioned Multi-Objective Alignment for Distributional Pareto-Optimality
  8. 2026MATO: Multi-objective Personalized Alignment with Test-time Optimization for Large Language Models