# Coherence Maximization Improves Pluralistic Alignment

arXiv: https://arxiv.org/abs/2606.03110 · HTML: https://mlsanity.com/paper/2606.03110/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Placed in Human values and AI feedback alignment, one of 10 directions in LLM alignment / RLHF / preference optimisation.
- Builds on 21 of its 31 references that are in the map.
- First posted on arXiv 2 Jun 2026.

## Placed in

- LLM alignment / RLHF / preference optimisation / [Human values and AI feedback alignment](https://mlsanity.com/alignment/human-values-and-ai-feedback-alignment/)

## Builds on (most cited first, 21 of 21)

- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [Constitutional AI: Harmlessness from AI Feedback](https://mlsanity.com/paper/2212.08073/) (2022)
- [Whose Opinions Do Language Models Reflect?](https://mlsanity.com/paper/2303.17548/) (2023)
- [Artificial Intelligence, Values and Alignment](https://mlsanity.com/paper/2001.09768/) (2020)
- [Persona Vectors: Monitoring and Controlling Character Traits in Language Models](https://mlsanity.com/paper/2507.21509/) (2025)
- [Principles alone cannot guarantee ethical AI](https://mlsanity.com/paper/1906.06668/) (2019)
- [STaR-GATE: Teaching Language Models to Ask Clarifying Questions](https://mlsanity.com/paper/2403.19154/) (2024)
- [Randomness, Not Representation: The Unreliability of Evaluating Cultural Alignment in LLMs](https://mlsanity.com/paper/2503.08688/) (2025)
- [Diverging Preferences: When do Annotators Disagree and do Models Know?](https://mlsanity.com/paper/2410.14632/) (2024)
- [What's In My Human Feedback? Learning Interpretable Descriptions of Preference Data](https://mlsanity.com/paper/2510.26202/) (2025)
- [Monitoring Decomposition Attacks in LLMs with Lightweight Sequential Monitors](https://mlsanity.com/paper/2506.10949/) (2025)
- [Open Character Training: Shaping the Persona of AI Assistants through Constitutional AI](https://mlsanity.com/paper/2511.01689/) (2025)
- [Inverse Reinforcement Learning with Dynamic Reward Scaling for LLM Alignment](https://mlsanity.com/paper/2503.18991/) (2025)
- [Self-Supervised Alignment with Mutual Information: Learning to Follow Principles without Preference Labels](https://mlsanity.com/paper/2404.14313/) (2024)
- [Whose Boat Does it Float? Improving Personalization in Preference Tuning via Inferred User Personas](https://mlsanity.com/paper/2501.11549/) (2025)
- [On the Alignment of Large Language Models with Global Human Opinion](https://mlsanity.com/paper/2509.01418/) (2025)
- [Generative Value Conflicts Reveal LLM Priorities](https://mlsanity.com/paper/2509.25369/) (2025)
- [Case Repositories: Towards Case-Based Reasoning for AI Alignment](https://mlsanity.com/paper/2311.10934/) (2023)
- [Statutory Construction and Interpretation for Artificial Intelligence](https://mlsanity.com/paper/2509.01186/) (2025)
- [Steerable Pluralism: Pluralistic Alignment via Few-Shot Comparative Regression](https://mlsanity.com/paper/2508.08509/) (2025)
- [Inference-Time Personalized Alignment with a Few User Preference Queries](https://mlsanity.com/paper/2511.02966/) (2025)

## Abstract (arXiv)

Aligning AI systems with diverse human values requires value specifications grounded in concrete examples, but generating such examples without extensive human supervision remains an open challenge. We investigate what makes these examples effective, using Internal Coherence Maximization (ICM) -- which infers labels by maximizing their mutual predictability -- to generate persona-specific examples that steer a model toward a target group's values, without human supervision. Across four benchmarks spanning classification, preference, and open-ended generation, ICM-inferred in-context examples match the performance of gold labels. Crucially, coherence matters beyond individual label accuracy: with accuracy held constant, more coherent examples generalize substantially better than incoherent ones. For personas underrepresented in pretraining data, targeted human feedback on the questions where the model is least certain about a persona's values yields better generalization than the same number of labels on arbitrary questions. These results identify coherence as a key design principle for scalable value specification, leveraging the diverse human perspectives already encoded in pretrained language models.