mlsanityData to 7 Oct 2026

Open Problems in Constitutional Preference Reconstruction

arXiv 2606.30116 ↗First posted 29 Jun 2026cs.AIcited by 0

On the map: LLM alignment / RLHF / preference optimisation → Human values and AI feedback alignment

Abstract (arXiv)

Pairwise preference data is widely used for training and evaluating language models (e.g., RLHF), but each datapoint records a choice, not the rationale behind it. Methods such as Inverse Constitutional AI (ICAI) attempt to improve interpretability by compressing datasets into short ``constitutions'' of natural-language principles. We argue this framing is under-specified: a flat list of principles is not yet an executable decision rule because it leaves principle composition implicit. We use the pairwise setting as a testbed to empirically characterize three open problems in constitutional methods. First, principle quality is hard to measure: coverage and accuracy are useful but incomplete proxies for end-to-end reconstruction. Second, composition is ambiguous: holding principles fixed, different executors (LLM judge versus majority vote) agree only 73% of the time. Third, constitutions differ between LLMs: cross-model vote agreement is 73%, whereas intra-model agreement is 81%. Across PRISM, AlpacaEval, and Chatbot Arena, we show that principle refinement (ICAI+) may be a first step towards ameliorating these problems: inter-executor agreement rises to 78%, and transparent executors match LLM judge accuracy (66% vs. 67%). Our results highlight that constitutions should be evaluated as constitution--executor systems, with implications for LLMs-as-a-judge broadly.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Human values and AI feedback alignment, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 17 of its 45 references that are in the map. First posted on arXiv 29 Jun 2026.

Builds on

17 of 45 references.

Closest ancestors · 7

Same direction, newest first

  1. 2026Democratic ICAI: Debating Our Way to Steering Principles from Preferences26 Jun
  2. 2025What's In My Human Feedback? Learning Interpretable Descriptions of Preference Data30 Oct
  3. 2024Collective Constitutional AI: Aligning a Language Model with Public Input12 Jun
  4. 2024Inverse Constitutional AI: Compressing Preferences into Principles2 Jun
  5. 2024The PRISM Alignment Dataset: What Participatory, Representative and Individualised Human Feedback Reveals About the Subjective and Multicultural Alignment of Large Language Models24 Apr
  6. 2024Chatbot Arena: An Open Platform for Evaluating LLMs by Human Preference7 Mar
  7. 2022Constitutional AI: Harmlessness from AI Feedback15 Dec

Other references · 10

  1. 2022Training language models to follow instructions with human feedbackcited by 15,712
  2. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  3. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,916
  4. 2017Deep reinforcement learning from human preferencescited by 3,846
  5. 2023G-Eval: NLG Evaluation using GPT-4 with Better Human Alignmentcited by 2,205
  6. 2025OpenAI GPT-5 System Cardcited by 910
  7. 2023Open Problems and Fundamental Limitations of Reinforcement Learning from Human Feedbackcited by 743
  8. 2024Prometheus 2: An Open Source Language Model Specialized in Evaluating Other Language Modelscited by 431
  9. 2025Rubrics as Rewards: Reinforcement Learning Beyond Verifiable Domainscited by 328
  10. 2024MT-Bench-101: A Fine-Grained Benchmark for Evaluating Large Language Models in Multi-Turn Dialoguescited by 247

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025Decoding Human Preferences in Alignment: An Improved Approach to Inverse Constitutional AI
  2. 2026Beyond Preferences: Learning Alignment Principles Grounded in Human Reasons and Values
  3. 2025From Delegates to Trustees: How Optimizing for Long-Term Interests Shapes Bias and Alignment in LLM
  4. 2026A Consensus-Based Framework for Relative Preference Evaluation of Large Language Models
  5. 2026Reflect: Transparent Principle-Guided Reasoning for Constitutional Alignment at Scale
  6. 2025C3AI: Crafting and Evaluating Constitutions for Constitutional AI
  7. 2025AMBEDKAR-A Multi-level Bias Elimination through a Decoding Approach with Knowledge Augmentation for Robust Constitutional Alignment of Language Models
  8. 2025Alignment Revisited: Are Large Language Models Consistent in Stated and Revealed Preferences?