mlsanityData to 14 Oct 2026

PARM: Multi-Objective Test-Time Alignment via Preference-Aware Autoregressive Reward Model

arXiv 2505.06274 ↗First posted 6 May 2025cs.LG cs.AIcited by 28

On the map: LLM alignment / RLHF / preference optimisation → LLM preference optimization methods

Abstract (arXiv)

Multi-objective test-time alignment aims to adapt large language models (LLMs) to diverse multi-dimensional user preferences during inference while keeping LLMs frozen. Recently, GenARM (Xu et al., 2025) first independently trains Autoregressive Reward Models (ARMs) for each preference dimension without awareness of each other, then combines their outputs based on user-specific preference vectors during inference to achieve multi-objective test-time alignment, leading to two key limitations: the need for textitmultiple ARMs increases the inference cost, and the separate training of ARMs causes the misalignment between the guided generation and the user preferences. To address these issues, we propose Preference-aware ARM (PARM), a single unified ARM trained across all preference dimensions. PARM uses our proposed Preference-Aware Bilinear Low-Rank Adaptation (PBLoRA), which employs a bilinear form to condition the ARM on preference vectors, enabling it to achieve precise control over preference trade-offs during inference. Experiments demonstrate that PARM reduces inference costs and achieves better alignment with preference vectors compared with existing methods. Additionally, PARM enables weak-to-strong guidance, allowing a smaller PARM to guide a larger frozen LLM without expensive training, making multi-objective alignment accessible with limited computing resources. The code is available at https://github.com/Baijiong-Lin/PARM.

Key facts. Cited by 28 papers in the map (data through 14 Oct 2026). Placed in LLM preference optimization methods, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 44 of its 52 references that are in the map. First posted on arXiv 6 May 2025.

Builds on

44 of 52 references.

Closest ancestors · 14

Same direction, newest first

  1. 2024GenARM: Reward Guided Generation with Autoregressive Reward Model for Test-time Alignment10 Oct
  2. 2024Decoding-Time Language Model Alignment with Multiple Objectives27 Jun
  3. 2024Cascade Reward Sampling for Efficient Decoding-Time Alignment24 Jun
  4. 2024Transfer Q Star: Principled Decoding for LLM Alignment30 May
  5. 2024SimPO: Simple Preference Optimization with a Reference-Free Reward23 May
  6. 2024Disentangling Length from Quality in Direct Preference Optimization28 Mar
  7. 2024MetaAligner: Towards Generalizable Multi-Objective Alignment of Language Models25 Mar
  8. 2024Controllable Preference Optimization: Toward Controllable Multi-Objective Alignment29 Feb
  9. 2024Arithmetic Control of LLMs for Diverse User Preferences: Directional Preference Alignment with Multi-Objective Rewards28 Feb
  10. 2024Rewards-in-Context: Multi-objective Alignment of Foundation Models with Dynamic Preference Adjustment15 Feb
  11. 2024DeAL: Decoding-time Alignment for Large Language Models5 Feb
  12. 2024Panacea: Pareto Alignment via Preference Adaptation for LLMs3 Feb
2 more
  1. 2023Beyond One-Preference-Fits-All Alignment: Multi-Objective Direct Preference Optimization5 Oct
  2. 2022Training language models to follow instructions with human feedback4 Mar

Other references · 30

  1. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  2. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 8,065
  3. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,720
  4. 2021Convex optimizationcited by 2,515
  5. 2023BeaverTails: Towards Improved Safety Alignment of LLM via a Human-Preference Datasetcited by 860
  6. 2017A Survey on Multi-Task Learningcited by 840
  7. 2019Plug and Play Language Models: A Simple Approach to Controlled Text Generationcited by 799
  8. 2023Open Problems and Fundamental Limitations of Reinforcement Learning from Human Feedbackcited by 759
  9. 2024TinyLlama: An Open-Source Small Language Modelcited by 618
  10. 2023Fine-Grained Human Feedback Gives Better Rewards for Language Model Trainingcited by 401
  11. 2023Personalized Soups: Personalized Large Language Model Alignment via Post-hoc Parameter Mergingcited by 280
  12. 2024PKU-SafeRLHF: Towards Multi-Level Safety Alignment for LLMs with Human Preferencecited by 261
18 more
  1. 2023Rewarded soups: towards Pareto-optimal alignment by interpolating weights fine-tuned on diverse rewardscited by 243
  2. 2021Reasonable Effectiveness of Random Weighting: A Litmus Test for Multi-Task Learningcited by 101
  3. 2024Controllable Text Generation for Large Language Models: A Surveycited by 57
  4. 2024Smooth Tchebycheff Scalarization for Multi-Objective Optimizationcited by 54
  5. 2022Pareto Set Learning for Expensive Multi-Objective Optimizationcited by 51
  6. 2023Controlled Text Generation via Language Model Arithmeticcited by 51
  7. 2024Conditional Language Policy: A General Framework for Steerable Multi-Objective Finetuningcited by 43
  8. 2021Multi-Objective Meta Learningcited by 39
  9. 2022Pareto Manifold Learning: Tackling multiple tasks via ensembles of single-task modelscited by 29
  10. 2023Dual-Balancing for Multi-Task Learningcited by 27
  11. 2025Pareto Set Learning for Multi-Objective Reinforcement Learningcited by 20
  12. 2025Gradient-Based Multi-Objective Deep Learning: Algorithms, Theories, Applications, and Beyondcited by 18
  13. 2024Pareto Merging: Multi-Objective Optimization for Preference-Aware Model Mergingcited by 14
  14. 2024Pareto Low-Rank Adapters: Efficient Multi-Task Learning with Preferencescited by 13
  15. 2024LibMOON: A Gradient-based MultiObjective OptimizatioN Library in PyTorchcited by 10
  16. 2024Efficient Pareto Manifold Learning with Low-Rank Structurecited by 9
  17. 2024FERERO: A Flexible Framework for Preference-Guided Multi-Objective Learningcited by 9
  18. 2024A First-Order Multi-Gradient Algorithm for Multi-Objective Bi-Level Optimizationcited by 8

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 10; 2026: 18.

28 papers.

Sort:

By area:

  1. 2025Multi-Objective Preference Optimization: Improving Human Alignment of Generative Modelscited by 13
  2. 2025Multi-objective Large Language Model Alignment with Hierarchical Expertscited by 13
  3. 2025MetaDefense: Defending Finetuning-based Jailbreak Attack Before and During Generationcited by 8
  4. 2025Simultaneous Multi-objective Alignment Across Verifiable and Non-verifiable Rewardscited by 6
  5. 2025PICACO: Pluralistic In-Context Value Alignment of LLMs via Total Correlation Optimizationcited by 3
  6. 2025ARCANE: A Multi-Agent Framework for Interpretable and Configurable Alignmentcited by 3
  7. 2026UniARM: Towards a Unified Autoregressive Reward Model for Multi-Objective Test-Time Alignmentcited by 3
  8. 2026RVPO: Risk-Sensitive Alignment via Variance Regularizationcited by 3
  9. 2025Precise Attribute Intensity Control in Large Language Models via Targeted Representation Editingcited by 2
  10. 2025ProSocialAlign: Preference Conditioned Test Time Alignment in Language Modelscited by 1
  11. 2026LLMdoctor: Token-Level Flow-Guided Preference Optimization for Efficient Test-Time Alignment of Large Language Modelscited by 1
  12. 2026FluxLite: Inference-Time Proposal Control for Discrete Diffusion Modelscited by 1
16 more
  1. 2025MAVIS: Multi-Objective Alignment via Inference-Time Value-Guided Selectioncited by 0
  2. 2025Multi-Objective Reward and Preference Optimization: Theory and Algorithmscited by 0
  3. 2026Bayesian Preference Learning for Test-Time Steerable Reward Modelscited by 0
  4. 2026Many Preferences, Few Policies: Towards Scalable Language Model Personalizationcited by 0
  5. 2026Pref-CTRL: Preference Driven LLM Alignment using Representation Editingcited by 0
  6. 2026MetaMoE: Diversity-Aware Proxy Selection for Privacy-Preserving Mixture-of-Experts Unificationcited by 0
  7. 2026SURF: Steering the Scalarization Weight to Uniformly Traverse the Pareto Frontcited by 0
  8. 2026MATO: Multi-objective Personalized Alignment with Test-time Optimization for Large Language Modelscited by 0
  9. 2026Accountable Human-AI Deliberation with LLMs: Scaling Collective Intelligence through Symbiotic Scaffoldingcited by 0
  10. 2026SPARD: Defending Harmful Fine-Tuning Attack via Safety Projection with Relevance-Diversity Data Selectioncited by 0
  11. 2026PAFO: Pareto Fairness Optimization for Personalized Reward Modelingcited by 0
  12. 2026Multi-Objective Exploration and Preference Optimization via Mutual Informationcited by 0
  13. 2026UniVR: Thinking in Visual Space for Unified Visual Reasoningcited by 0
  14. 2026Inference-Time Policy Alignment for Fair Reinforcement Learningcited by 0
  15. 2026Trust Your Guide Only When Certain: Uncertainty-Aware Sparse Alignment at Inference Timecited by 0
  16. 2026The Missing Coefficients: Bayesian Pairwise Merging for Model Personalizationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Preference Orchestrator: Prompt-Aware Multi-Objective Alignment for Large Language Models
  2. 2026Meta-Aligner: Bidirectional Preference-Policy Optimization for Multi-Objective LLMs Alignment
  3. 2025Test-Time Preference Optimization: On-the-Fly Alignment via Iterative Textual Feedback
  4. 2024Inference time LLM alignment in single and multidomain preference spectrum
  5. 2025PITA: Preference-Guided Inference-Time Alignment for LLM Post-Training
  6. 2024Few-shot Steerable Alignment: Adapting Rewards and LLM Policies with Neural Processes
  7. 2026Reward-free Alignment for Conflicting Objectives
  8. 2025UC-MOA: Utility-Conditioned Multi-Objective Alignment for Distributional Pareto-Optimality