mlsanityData to 7 Oct 2026

Self-Supervised Alignment with Mutual Information: Learning to Follow Principles without Preference Labels

arXiv 2404.14313 ↗First posted 22 Apr 2024cs.CLcited by 18

On the map: LLM alignment / RLHF / preference optimisation → LLM preference optimization methods

Abstract (arXiv)

When prompting a language model (LM), users often expect the model to adhere to a set of behavioral principles across diverse tasks, such as producing insightful content while avoiding harmful or biased language. Instilling such principles (i.e., a constitution) into a model is resource-intensive, technically challenging, and generally requires human preference labels or examples. We introduce SAMI, an iterative algorithm that finetunes a pretrained language model (without requiring preference labels or demonstrations) to increase the conditional mutual information between constitutions and self-generated responses given queries from a dataset. On single-turn dialogue and summarization, a SAMI-trained mistral-7b outperforms the initial pretrained model, with win rates between 66% and 77%. Strikingly, it also surpasses an instruction-finetuned baseline (mistral-7b-instruct) with win rates between 55% and 57% on single-turn dialogue. SAMI requires a model that writes the principles. To avoid dependence on strong models for writing principles, we align a strong pretrained model (mixtral-8x7b) using constitutions written by a weak instruction-finetuned model (mistral-7b-instruct), achieving a 65% win rate on summarization. Finally, we investigate whether SAMI generalizes to diverse summarization principles (e.g., "summaries should be scientific") and scales to stronger models (llama3-70b), finding that it achieves win rates of up to 68% for learned and 67% for held-out principles compared to the base model. Our results show that a pretrained LM can learn to follow constitutions without using preference labels, demonstrations, or human oversight.

Key facts. Cited by 18 papers in the map (data through 7 Oct 2026). Placed in LLM preference optimization methods, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 34 of its 48 references that are in the map. First posted on arXiv 22 Apr 2024.

Builds on

34 of 48 references.

Closest ancestors · 8

Same direction, newest first

  1. 2024Disentangling Length from Quality in Direct Preference Optimization28 Mar
  2. 2024Relative Preference Optimization: Enhancing LLM Alignment through Contrasting Responses across Identical and Diverse Prompts12 Feb
  3. 2024Generalized Preference Optimization: A Unified Approach to Offline Alignment8 Feb
  4. 2023The Unlocking Spell on Base LLMs: Rethinking Alignment via In-Context Learning4 Dec
  5. 2023A General Theoretical Paradigm to Understand Learning from Human Preferences18 Oct
  6. 2023Principle-Driven Self-Alignment of Language Models from Scratch with Minimal Human Supervision4 May
  7. 2022Training language models to follow instructions with human feedback4 Mar
  8. 2021A General Language Assistant as a Laboratory for Alignment1 Dec

Other references · 26

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2023GPT-4 Technical Reportcited by 18,132
  3. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  4. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  5. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  6. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,916
  7. 2018Representation Learning with Contrastive Predictive Codingcited by 7,662
  8. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,767
  9. 2017Deep reinforcement learning from human preferencescited by 3,846
  10. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,634
  11. 2023Mistral 7Bcited by 3,065
  12. 2022Constitutional AI: Harmlessness from AI Feedbackcited by 2,950
14 more
  1. 2020Learning to summarize from human feedbackcited by 2,562
  2. 2024Mixtral of Expertscited by 1,785
  3. 2023Large Language Models are not Fair Evaluatorscited by 1,050
  4. 2022STaR: Bootstrapping Reasoning With Reasoningcited by 895
  5. 2019On Variational Bounds of Mutual Informationcited by 550
  6. 2022Improving alignment of dialogue agents via targeted human judgementscited by 527
  7. 2023Weak-to-Strong Generalization: Eliciting Strong Capabilities With Weak Supervisioncited by 445
  8. 2023Towards Measuring the Representation of Subjective Global Opinions in Language Modelscited by 424
  9. 2023SLiC-HF: Sequence Likelihood Calibration with Human Feedbackcited by 395
  10. 2023Toxicity in ChatGPT: Analyzing Persona-assigned Language Modelscited by 356
  11. 2017Thinking Fast and Slow with Deep Learning and Tree Searchcited by 245
  12. 2024STaR-GATE: Teaching Language Models to Ask Clarifying Questionscited by 99
  13. 2023Specific versus General Principles for Constitutional AIcited by 41
  14. 2023Social Contract AI: Aligning AI Assistants with Implicit Group Normscited by 9

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 8; 2025: 6; 2026: 4.

18 papers.

Sort:

By area:

  1. 2024Generative Reward Modelscited by 148
  2. 2024PERSONA: A Reproducible Testbed for Pluralistic Alignmentcited by 71
  3. 2024Enhancing Large Vision Language Models with Self-Training on Image Comprehensioncited by 70
  4. 2024WPO: Enhancing RLHF with Weighted Preference Optimizationcited by 53
  5. 2025SeRL: Self-Play Reinforcement Learning for Large Language Models with Limited Datacited by 50
  6. 2025An Information Theoretic Perspective on Agentic System Designcited by 8
  7. 2025Stop Reducing Responsibility in LLM-Powered Multi-Agent Systems to Local Alignmentcited by 7
  8. 2025LLM Safety Alignment is Divergence Estimation in Disguisecited by 6
  9. 2024Self-Steering Optimization: Autonomous Preference Optimization for Large Language Modelscited by 5
  10. 2024Is Free Self-Alignment Possible?cited by 4
  11. 2024Self-Control of LLM Behaviors by Compressing Suffix Gradient into Prefix Controllercited by 3
  12. 2025ENIGMA: The Geometry of Reasoning and Alignment in Large-Language Modelscited by 1
6 more
  1. 2026Reflect: Transparent Principle-Guided Reasoning for Constitutional Alignment at Scalecited by 1
  2. 2026Generating Data-Driven Reasoning Rubrics for Domain-Adaptive Reward Modelingcited by 1
  3. 2024An Exploration of Self-Supervised Mutual Information Alignment for Multi-Task Settingscited by 0
  4. 2025Latent Principle Discovery for Language Model Self-Improvementcited by 0
  5. 2026Maximizing Mutual Information Between Prompt and Response Improves LLM Performance With No Additional Datacited by 0
  6. 2026Coherence Maximization Improves Pluralistic Alignmentcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Aligning Large Language Models via Fully Self-Synthetic Data
  2. 2024Improving the Robustness of Large Language Models via Consistency Alignment
  3. 2025P-Aligner: Enabling Pre-Alignment of Language Models via Principled Instruction Synthesis
  4. 2025Stackelberg Self-Annotation: A Robust Approach to Data-Efficient LLM Alignment
  5. 2025Icon2: Aligning Large Language Models Using Self-Synthetic Preference Data via Inherent Regulation
  6. 2023CycleAlign: Iterative Distillation from Black-box LLM to White-box Models for Better Human Alignment
  7. 2023ULMA: Unified Language Model Alignment with Human Demonstration and Point-wise Preference
  8. 2024Human-Instruction-Free LLM Self-Alignment with Limited Samples