mlsanityData to 14 Oct 2026

Decoding Human Preferences in Alignment: An Improved Approach to Inverse Constitutional AI

arXiv 2501.17112 ↗First posted 28 Jan 2025cs.LGcited by 4

On the map: LLM alignment / RLHF / preference optimisation → Human values and AI feedback alignment

Abstract (arXiv)

Traditional methods for aligning Large Language Models (LLMs), such as Reinforcement Learning from Human Feedback (RLHF) and Direct Preference Optimization (DPO), rely on implicit principles, limiting interpretability. Constitutional AI (CAI) offers an explicit, rule-based framework for guiding LLM alignment. Building on this, we refine the Inverse Constitutional AI (ICAI) algorithm, which extracts constitutions from preference datasets. By improving principle generation, clustering, and embedding processes, our approach enhances the accuracy and generalizability of extracted principles across synthetic and real-world datasets. Our results highlight the potential of these principles to foster more transparent and adaptable alignment methods, offering a promising direction for future advancements beyond traditional fine-tuning.

Key facts. Cited by 4 papers in the map (data through 14 Oct 2026). Placed in Human values and AI feedback alignment, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 9 of its 12 references that are in the map. First posted on arXiv 28 Jan 2025.

Builds on

9 of 12 references.

Closest ancestors · 3

Same direction, newest first

  1. 2024Inverse Constitutional AI: Compressing Preferences into Principles2 Jun
  2. 2024Aligning LLM Agents by Learning Latent Preference from User Edits23 Apr
  3. 2023UltraFeedback: Boosting Language Models with Scaled AI Feedback2 Oct

Other references · 6

  1. 2019DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lightercited by 4,307
  2. 2024Length-Controlled AlpacaEval: A Simple Way to Debias Automatic Evaluatorscited by 897
  3. 2023BeaverTails: Towards Improved Safety Alignment of LLM via a Human-Preference Datasetcited by 860
  4. 2023Principle-Driven Self-Alignment of Language Models from Scratch with Minimal Human Supervisioncited by 318
  5. 2022Same Author or Just Same Topic? Towards Content-Independent Style Representationscited by 86
  6. 2019A Repository of Conversational Datasetscited by 45

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 1; 2026: 3.

4 papers.

  1. 2025Distributional AGI Safetycited by 12
  2. 2026Democratic ICAI: Debating Our Way to Steering Principles from Preferencescited by 1
  3. 2026Alignment Plausibility: A New Standard for Assuring AI in Healthcarecited by 0
  4. 2026Corpus Characterization and Inverse Constitutional Fine-Tuning for Style-Aware Radiology Reportscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Beyond Preferences: Learning Alignment Principles Grounded in Human Reasons and Values
  2. 2024Linear Alignment: A Closed-form Solution for Aligning Human Preferences without Tuning and Feedback
  3. 2026Open Problems in Constitutional Preference Reconstruction
  4. 2024Supervised Fine-Tuning as Inverse Reinforcement Learning
  5. 2023CycleAlign: Iterative Distillation from Black-box LLM to White-box Models for Better Human Alignment
  6. 2025HPS: Hard Preference Sampling for Human Preference Alignment
  7. 2026From RLHF to Direct Alignment: A Theoretical Unification of Preference Learning for Large Language Models
  8. 2024Panacea: Pareto Alignment via Preference Adaptation for LLMs