mlsanityData to 7 Oct 2026

From Constitutions to Control: Interpretable Rewards for Aligning Language Models

arXiv 2609.33086 ↗First posted 27 Sep 2026cs.LGcited by 0

On the map: LLM alignment / RLHF / preference optimisation → Human values and AI feedback alignment

Abstract (arXiv)

Current approaches to aligning language models often make it hard to know what behavior is being rewarded or to change that reward in a targeted way. In particular, standard preference-based methods collapse multiple considerations into aggregate human judgments, obscuring what drives the resulting reward, while principle-based methods specify high-level values without fully operationalizing them. To address this gap, we develop a rubric-based framework to transform a general-purpose constitution into an interpretable and tunable reward model, using constitution-guided AI feedback to estimate initial weights for the constituent rubric items. We then reweight those dimensions to construct modified rewards for training. Across experiments on political alignment and safety-helpfulness tradeoffs, reweighting individual dimensions predictably changes targeted behaviors largely independently while navigating tradeoffs between conflicting alignment objectives. We show that the same framework can mitigate label bias encoded in preference judgments -- including sycophancy and demographic bias -- by reducing their influence on the training reward. Our results demonstrate that constitution-derived, interpretable rewards can translate high-level alignment principles into more transparent and controllable model behavior.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Human values and AI feedback alignment, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 23 of its 79 references that are in the map. First posted on arXiv 27 Sep 2026.

Builds on

23 of 79 references.

Closest ancestors · 1

Same direction, newest first

  1. 2022Constitutional AI: Harmlessness from AI Feedback15 Dec

Other references · 22

  1. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  2. 2025Qwen3 Technical Reportcited by 8,195
  3. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,634
  4. 2025Gemma 3 Technical Reportcited by 1,686
  5. 2025gpt-oss-120b & gpt-oss-20b Model Cardcited by 1,258
  6. 2023Instruction-Following Evaluation for Large Language Modelscited by 1,143
  7. 2024Tulu 3: Pushing Frontiers in Open Language Model Post-Trainingcited by 988
  8. 2024Rewards-in-Context: Multi-objective Alignment of Foundation Models with Dynamic Preference Adjustmentcited by 171
  9. 2025Reinforcement Learning with Rubric Anchorscited by 108
  10. 2024Quantile Regression for Distributional Reward Models in RLHFcited by 59
  11. 2025Beyond Reward Hacking: Causal Rewards for Large Language Model Alignmentcited by 42
  12. 2025Auto-Rubric: Learning From Implicit Weights to Explicit Rubrics for Reward Modelingcited by 29
10 more
  1. 2025QA-LIGN: Aligning LLMs through Constitutionally Decomposed QAcited by 28
  2. 2026How RLHF Amplifies Sycophancycited by 20
  3. 2026Reward Hacking in Rubric-Based Reinforcement Learningcited by 18
  4. 2025Debiasing Reward Models by Representation Learning with Guaranteescited by 6
  5. 2025Interpretable Reward Modeling with Active Concept Bottleneckscited by 3
  6. 2026Reward Bias Substitution: Single-Axis Bias Mitigations Redirect Optimization Pressurecited by 3
  7. 2026SteerRM: Debiasing Reward Models via Sparse Autoencoderscited by 2
  8. 2026Mitigating Label Bias with Interpretable Rubric Embeddingscited by 1
  9. 2026Affective Context Amplifies Sycophancy in LLM Responsescited by 1
  10. 2026Receptiveness, Not Sycophancy: Distinguishing Engagement from Deference in Language Modelscited by 1

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025Targeting Misalignment: A Conflict-Aware Framework for Reward-Model-based LLM Alignment
  2. 2024Transforming and Combining Rewards for Aligning Large Language Models
  3. 2024Elephant in the Room: Unveiling the Impact of Reward Model Quality in Alignment
  4. 2025Alignment is Localized: A Causal Probe into Preference Layers
  5. 2026Misaligned by Reward: Socially Undesirable Preferences in LLMs
  6. 2026Alignment Makes Language Models Normative, Not Descriptive
  7. 2026The Neutral Mask: How Alignment Training Provides Shallow Alignment while Leaving Partisan Structure Intact in a Large Language Model
  8. 2025Capturing Nuanced Preferences: Preference-Aligned Distillation for Small Language Models