# STAGE: Controlled Objective Admission for Multi-Preference LLM Alignment

arXiv: https://arxiv.org/abs/2608.16553 · HTML: https://mlsanity.com/paper/2608.16553/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Placed in LLM preference optimization methods, one of 10 directions in LLM alignment / RLHF / preference optimisation.
- Builds on 23 of its 39 references that are in the map.
- First posted on arXiv 17 Aug 2026.

## Placed in

- LLM alignment / RLHF / preference optimisation / [LLM preference optimization methods](https://mlsanity.com/alignment/llm-preference-optimization-methods/)

## Builds on (most cited first, 23 of 23)

- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [Deep reinforcement learning from human preferences](https://mlsanity.com/paper/1706.03741/) (2017)
- [TruthfulQA: Measuring How Models Mimic Human Falsehoods](https://mlsanity.com/paper/2109.07958/) (2021)
- [Learning to summarize from human feedback](https://mlsanity.com/paper/2009.01325/) (2020)
- [Gradient Surgery for Multi-Task Learning](https://mlsanity.com/paper/2001.06782/) (2020)
- [Multi-Task Learning as Multi-Objective Optimization](https://mlsanity.com/paper/1810.04650/) (2018)
- [Automated Hate Speech Detection and the Problem of Offensive Language](https://mlsanity.com/paper/1703.04009/) (2017)
- [Fine-Grained Human Feedback Gives Better Rewards for Language Model Training](https://mlsanity.com/paper/2306.01693/) (2023)
- [PKU-SafeRLHF: Towards Multi-Level Safety Alignment for LLMs with Human Preference](https://mlsanity.com/paper/2406.15513/) (2024)
- [OR-Bench: An Over-Refusal Benchmark for Large Language Models](https://mlsanity.com/paper/2405.20947/) (2024)
- [Rewarded soups: towards Pareto-optimal alignment by interpolating weights fine-tuned on diverse rewards](https://mlsanity.com/paper/2306.04488/) (2023)
- [Reward Model Ensembles Help Mitigate Overoptimization](https://mlsanity.com/paper/2310.02743/) (2023)
- [Rewards-in-Context: Multi-objective Alignment of Foundation Models with Dynamic Preference Adjustment](https://mlsanity.com/paper/2402.10207/) (2024)
- [Controllable Preference Optimization: Toward Controllable Multi-Objective Alignment](https://mlsanity.com/paper/2402.19085/) (2024)
- [Why Should Adversarial Perturbations be Imperceptible? Rethink the Research Paradigm in Adversarial NLP](https://mlsanity.com/paper/2210.10683/) (2022)
- [NaturalReasoning: Reasoning in the Wild with 2.8M Challenging Questions](https://mlsanity.com/paper/2502.13124/) (2025)
- [Curry-DPO: Enhancing Alignment using Curriculum Learning & Ranked Preferences](https://mlsanity.com/paper/2403.07230/) (2024)
- [Pareto Multi-Objective Alignment for Language Models](https://mlsanity.com/paper/2508.07768/) (2025)
- [Understanding and Mitigating Overrefusal in LLMs from an Unveiling Perspective of Safety Decision Boundary](https://mlsanity.com/paper/2505.18325/) (2025)
- [Satirical News Detection and Analysis using Attention Mechanism and Linguistic Features](https://mlsanity.com/paper/1709.01189/) (2017)
- [Multi-objective Reinforcement learning from AI Feedback](https://mlsanity.com/paper/2406.07295/) (2024)
- [REWARD CONSISTENCY: Improving Multi-Objective Alignment from a Data-Centric Perspective](https://mlsanity.com/paper/2504.11337/) (2025)

## Built on by (most cited first, 1 of 1)

- [Ready2Blend: From Natural-Language Instructions to Composable Alignment Prompts](https://mlsanity.com/paper/2609.39365/) (2026)

## Abstract (arXiv)

Multi-preference alignment is often framed as scalarization: combine reward dimensions, then optimize. This leaves a temporal decision underspecified: when should each preference dimension enter policy optimization? We propose methodname, a stability-guided active-set controller for controlled objective admission. methodname starts from a small active set, retains admitted objectives, and expands when reward-deviation gates indicate low recent deviation or a patience budget is exhausted. A probing phase estimates a hard-to-easy order, and adaptive weighting emphasizes underperforming active dimensions. Automatic evaluations with 15 training preferences and 16 held-out benchmark columns show that methodname obtains higher averages than simultaneous scalarization and shared-budget adapted baselines. Component ablations and expansion dynamics further support cumulative retention, gated admission, and probing-derived ordering as useful design choices in this setting. These results position objective-entry timing as a concrete control variable in reward-vector RLHF.