mlsanityData to 14 Oct 2026

MAVIS: Multi-Objective Alignment via Inference-Time Value-Guided Selection

arXiv 2508.13415 ↗First posted 19 Aug 2025cs.LGcited by 0

On the map: LLM alignment / RLHF / preference optimisation → LLM preference optimization methods

Abstract (arXiv)

Large Language Models (LLMs) are increasingly deployed across diverse applications that demand balancing multiple, often conflicting, objectives -- such as helpfulness, harmlessness, or humor. Many traditional methods for aligning outputs to user-specific preferences require fine-tuning models for each objective or for specific preference configurations, which is computationally expensive and inflexible. We introduce MAVIS -- textitMulti-Objective Alignment via Inference-Time Value-Guided Selection -- a lightweight inference-time alignment framework that enables dynamic control over LLM behavior without modifying the base model's weights. MAVIS trains a set of small value models, each corresponding to a distinct objective. At inference time, these value models are combined using user-specified weights to produce a tilting function that adjusts the base model's output distribution toward desired trade-offs. The value models are trained using a simple iterative algorithm that enables monotonic improvement of the KL-regularized policy. We show empirically that MAVIS achieves a superior pareto front compared to baselines which fine-tune per-objective models and combine them post hoc or train a single preference-conditioned value model for guidance. Our code is available at https://github.com/5-Jeremy/MAVIS/tree/main.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in LLM preference optimization methods, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 42 of its 56 references that are in the map. First posted on arXiv 19 Aug 2025.

Builds on

42 of 56 references.

Closest ancestors · 16

Same direction, newest first

  1. 2025Bounded Rationality for LLMs: Satisficing Alignment at Inference-Time29 May
  2. 2025Multi-Objective Preference Optimization: Improving Human Alignment of Generative Models16 May
  3. 2025PARM: Multi-Objective Test-Time Alignment via Preference-Aware Autoregressive Reward Model6 May
  4. 2025Robust Multi-Objective Preference Alignment with Online DPO1 Mar
  5. 2025On Almost Surely Safe Alignment of Large Language Models at Inference-Time3 Feb
  6. 2024GenARM: Reward Guided Generation with Autoregressive Reward Model for Test-time Alignment10 Oct
  7. 2024Inference-Time Language Model Alignment via Integrated Value Guidance26 Sep
  8. 2024Q-Adapter: Customizing Pre-trained LLMs to New Preferences with Forgetting Mitigation4 Jul
  9. 2024Decoding-Time Language Model Alignment with Multiple Objectives27 Jun
  10. 2024Aligning Large Language Models with Representation Editing: A Control Perspective10 Jun
  11. 2024Transfer Q Star: Principled Decoding for LLM Alignment30 May
  12. 2024Value Augmented Sampling for Language Model Alignment and Personalization10 May
4 more
  1. 2024Rewards-in-Context: Multi-objective Alignment of Foundation Models with Dynamic Preference Adjustment15 Feb
  2. 2024Direct Language Model Alignment from Online AI Feedback7 Feb
  3. 2024Linear Alignment: A Closed-form Solution for Aligning Human Preferences without Tuning and Feedback21 Jan
  4. 2022Training language models to follow instructions with human feedback4 Mar

Other references · 26

  1. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 8,065
  2. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,885
  3. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,720
  4. 2020Learning to summarize from human feedbackcited by 2,588
  5. 2022Scaling Laws for Reward Model Overoptimizationcited by 1,095
  6. 2018Adafactor: Adaptive Learning Rates with Sublinear Memory Costcited by 905
  7. 2021FUDGE: Controlled Text Generation With Future Discriminatorscited by 376
  8. 2023Alphazero-like Tree-Search can Guide Large Language Model Decoding and Trainingcited by 295
  9. 2024PKU-SafeRLHF: Towards Multi-Level Safety Alignment for LLMs with Human Preferencecited by 261
  10. 2023Rewarded soups: towards Pareto-optimal alignment by interpolating weights fine-tuned on diverse rewardscited by 243
  11. 2024VinePPO: Refining Credit Assignment in RL Training of LLMscited by 183
  12. 2019A Generalized Algorithm for Multi-Objective Reinforcement Learning and Policy Adaptationcited by 177
14 more
  1. 2018Deep Reinforcement Learning and the Deadly Triadcited by 174
  2. 2022Offline RL for Natural Language Generation with Implicit Language Q Learningcited by 138
  3. 2021Improving Faithfulness in Abstractive Summarization with Contrast Candidate Generation and Selectioncited by 76
  4. 2023Don't throw away your value model! Generating more preferable text with Value-Guided Monte-Carlo Tree Search decodingcited by 60
  5. 2024Conditional Language Policy: A General Framework for Steerable Multi-Objective Finetuningcited by 43
  6. 2024Entropy-Regularized Process Reward Modelcited by 38
  7. 2025Amulet: ReAlignment During Test Time for Personalized Preference Adaptation of LLMscited by 29
  8. 2025QLASS: Boosting Language Agent Inference via Q-Guided Stepwise Searchcited by 20
  9. 2022Anchor-Changing Regularized Natural Policy Gradient for Multi-Objective Reinforcement Learningcited by 15
  10. 2025Qsharp: Provably Optimal Distributional RL for LLM Post-Trainingcited by 15
  11. 2024VerifierQ: Enhancing LLM Test Time Compute with Q-Learning-based Verifierscited by 14
  12. 2025Value-Guided Search for Efficient Chain-of-Thought Reasoningcited by 13
  13. 2025Robust Multi-Objective Controlled Decoding of Large Language Modelscited by 9
  14. 2025Towards Cost-Effective Reward Guided Text Generationcited by 5

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025Preference Orchestrator: Prompt-Aware Multi-Objective Alignment for Large Language Models
  2. 2024MetaAligner: Towards Generalizable Multi-Objective Alignment of Language Models
  3. 2025Simultaneous Multi-objective Alignment Across Verifiable and Non-verifiable Rewards
  4. 2024Multi-Objective Alignment of Large Language Models Through Hypervolume Maximization
  5. 2025Multi-objective Large Language Model Alignment with Hierarchical Experts
  6. 2026Reward-free Alignment for Conflicting Objectives
  7. 2026Meta-Aligner: Bidirectional Preference-Policy Optimization for Multi-Objective LLMs Alignment
  8. 2024Panacea: Pareto Alignment via Preference Adaptation for LLMs