# Decoding Human Preferences in Alignment: An Improved Approach to Inverse Constitutional AI

arXiv: https://arxiv.org/abs/2501.17112 · HTML: https://mlsanity.com/paper/2501.17112/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Placed in Human values and AI feedback alignment, one of 10 directions in LLM alignment / RLHF / preference optimisation.
- Builds on 9 of its 12 references that are in the map.
- First posted on arXiv 28 Jan 2025.

## Placed in

- LLM alignment / RLHF / preference optimisation / [Human values and AI feedback alignment](https://mlsanity.com/alignment/human-values-and-ai-feedback-alignment/)

## Builds on (most cited first, 9 of 9)

- [DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter](https://mlsanity.com/paper/1910.01108/) (2019)
- [Length-Controlled AlpacaEval: A Simple Way to Debias Automatic Evaluators](https://mlsanity.com/paper/2404.04475/) (2024)
- [BeaverTails: Towards Improved Safety Alignment of LLM via a Human-Preference Dataset](https://mlsanity.com/paper/2307.04657/) (2023)
- [UltraFeedback: Boosting Language Models with Scaled AI Feedback](https://mlsanity.com/paper/2310.01377/) (2023)
- [Principle-Driven Self-Alignment of Language Models from Scratch with Minimal Human Supervision](https://mlsanity.com/paper/2305.03047/) (2023)
- [Same Author or Just Same Topic? Towards Content-Independent Style Representations](https://mlsanity.com/paper/2204.04907/) (2022)
- [Aligning LLM Agents by Learning Latent Preference from User Edits](https://mlsanity.com/paper/2404.15269/) (2024)
- [A Repository of Conversational Datasets](https://mlsanity.com/paper/1904.06472/) (2019)
- [Inverse Constitutional AI: Compressing Preferences into Principles](https://mlsanity.com/paper/2406.06560/) (2024)

## Built on by (most cited first, 4 of 4)

- [Distributional AGI Safety](https://mlsanity.com/paper/2512.16856/) (2025)
- [Democratic ICAI: Debating Our Way to Steering Principles from Preferences](https://mlsanity.com/paper/2606.28294/) (2026)
- [Alignment Plausibility: A New Standard for Assuring AI in Healthcare](https://mlsanity.com/paper/2607.07766/) (2026)
- [Corpus Characterization and Inverse Constitutional Fine-Tuning for Style-Aware Radiology Reports](https://mlsanity.com/paper/2609.14226/) (2026)

## Abstract (arXiv)

Traditional methods for aligning Large Language Models (LLMs), such as Reinforcement Learning from Human Feedback (RLHF) and Direct Preference Optimization (DPO), rely on implicit principles, limiting interpretability. Constitutional AI (CAI) offers an explicit, rule-based framework for guiding LLM alignment. Building on this, we refine the Inverse Constitutional AI (ICAI) algorithm, which extracts constitutions from preference datasets. By improving principle generation, clustering, and embedding processes, our approach enhances the accuracy and generalizability of extracted principles across synthetic and real-world datasets. Our results highlight the potential of these principles to foster more transparent and adaptable alignment methods, offering a promising direction for future advancements beyond traditional fine-tuning.