mlsanityData to 7 Oct 2026

Aligning Large Language Models from Self-Reference AI Feedback with one General Principle

arXiv 2406.11190 ↗First posted 17 Jun 2024cs.CL cs.AIcited by 2

On the map: LLM alignment / RLHF / preference optimisation → Human values and AI feedback alignment

Abstract (arXiv)

In aligning large language models (LLMs), utilizing feedback from existing advanced AI rather than humans is an important method to scale supervisory signals. However, it is highly challenging for AI to understand human intentions and societal values, and provide accurate preference feedback based on these. Current AI feedback methods rely on powerful LLMs, carefully designed specific principles to describe human intentions, and are easily influenced by position bias. To address these issues, we propose a self-reference-based AI feedback framework that enables a 13B Llama2-Chat to provide high-quality feedback under simple and general principles such as ``best for humanity``. Specifically, we allow the AI to first respond to the user's instructions, then generate criticism of other answers based on its own response as a reference, and finally determine which answer better fits human preferences according to the criticism. Additionally, we use a self-consistency method to further reduce the impact of position bias, and employ semantic perplexity to calculate the preference strength differences between different answers. Experimental results show that our method enables 13B and 70B Llama2-Chat annotators to provide high-quality preference feedback, and the policy models trained based on these preference data achieve significant advantages in benchmark datasets through reinforcement learning.

Key facts. Cited by 2 papers in the map (data through 7 Oct 2026). Placed in Human values and AI feedback alignment, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 29 of its 38 references that are in the map. First posted on arXiv 17 Jun 2024.

Builds on

29 of 38 references.

Closest ancestors · 3

Same direction, newest first

  1. 2023Specific versus General Principles for Constitutional AI20 Oct
  2. 2023AlpacaFarm: A Simulation Framework for Methods that Learn from Human Feedback22 May
  3. 2020Learning to summarize from human feedback2 Sep

Other references · 26

  1. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  2. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  3. 2017Deep reinforcement learning from human preferencescited by 3,846
  4. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,747
  5. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,634
  6. 2023G-Eval: NLG Evaluation using GPT-4 with Better Human Alignmentcited by 2,205
  7. 2019Fine-Tuning Language Models from Human Preferencescited by 2,124
  8. 2015High-Dimensional Continuous Control Using Generalized Advantage Estimationcited by 1,916
  9. 2023FActScore: Fine-grained Atomic Evaluation of Factual Precision in Long Form Text Generationcited by 1,259
  10. 2023Large Language Models are not Fair Evaluatorscited by 1,050
  11. 2021BARTScore: Evaluating Generated Text as Text Generationcited by 719
  12. 2023PyTorch FSDP: Experiences on Scaling Fully Sharded Data Parallelcited by 641
14 more
  1. 2023Safe RLHF: Safe Reinforcement Learning from Human Feedbackcited by 630
  2. 2023RLAIF vs. RLHF: Scaling Reinforcement Learning from Human Feedback with AI Feedbackcited by 615
  3. 2022Improving alignment of dialogue agents via targeted human judgementscited by 527
  4. 2023Secrets of RLHF in Large Language Models Part I: PPOcited by 226
  5. 2024Secrets of RLHF in Large Language Models Part II: Reward Modelingcited by 170
  6. 2022Language Models with Image Descriptors are Strong Few-Shot Video-Language Learnerscited by 117
  7. 2020BLEU might be Guilty but References are not Innocentcited by 109
  8. 2023Exploring the Use of Large Language Models for Reference-Free Text Quality Evaluation: An Empirical Studycited by 85
  9. 2024LoRA Land: 310 Fine-tuned LLMs that Rival GPT-4, A Technical Reportcited by 47
  10. 2023Multi-Dimensional Evaluation of Text Summarization with In-Context Learningcited by 44
  11. 2023Closing the Curious Case of Neural Text Degenerationcited by 30
  12. 2023Improving Generalization of Alignment with Human Preferences through Group Invariant Learningcited by 25
  13. 2023Concrete Problems in AI Safety, Revisitedcited by 21
  14. 2023ALLURE: Auditing and Improving LLM-based Evaluation of Text using Iterative In-Context-Learningcited by 15

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 1; 2026: 1.

2 papers.

  1. 2025Alignment and Safety in Large Language Models: Safety Mechanisms, Training Paradigms, and Emerging Challengescited by 14
  2. 2026Unifying Stable Optimization and Reference Regularization in RLHFcited by 4

Similar, not cited

Close in wording, no citation either way

  1. 2025Aligning Large Language Models via Fully Self-Synthetic Data
  2. 2023Peering Through Preferences: Unraveling Feedback Acquisition for Aligning Large Language Models
  3. 2023Data-Efficient Alignment of Large Language Models with Human Feedback Through Natural Language
  4. 2023Principle-Driven Self-Alignment of Language Models from Scratch with Minimal Human Supervision
  5. 2025Human Preferences for Constructive Interactions in Language Model Alignment
  6. 2024A Survey on Human Preference Learning for Large Language Models
  7. 2026LLMs Can Better Capture Human Judgments--With the Right Prompts
  8. 2025DxHF: Providing High-Quality Human Feedback for LLM Alignment via Interactive Decomposition