mlsanityData to 7 Oct 2026

Self-Evolution Fine-Tuning for Policy Optimization

arXiv 2406.10813 ↗First posted 16 Jun 2024cs.CLcited by 2

On the map: LLM alignment / RLHF / preference optimisation → LLM preference optimization methods

Abstract (arXiv)

The alignment of large language models (LLMs) is crucial not only for unlocking their potential in specific tasks but also for ensuring that responses meet human expectations and adhere to safety and ethical principles. Current alignment methodologies face considerable challenges. For instance, supervised fine-tuning (SFT) requires extensive, high-quality annotated samples, while reinforcement learning from human feedback (RLHF) is complex and often unstable. In this paper, we introduce self-evolution fine-tuning (SEFT) for policy optimization, with the aim of eliminating the need for annotated samples while retaining the stability and efficiency of SFT. SEFT first trains an adaptive reviser to elevate low-quality responses while maintaining high-quality ones. The reviser then gradually guides the policy's optimization by fine-tuning it with enhanced responses. One of the prominent features of this method is its ability to leverage unlimited amounts of unannotated data for policy optimization through supervised fine-tuning. Our experiments on AlpacaEval 2.0 and MT-Bench demonstrate the effectiveness of SEFT. We also provide a comprehensive analysis of its advantages over existing alignment techniques.

Key facts. Cited by 2 papers in the map (data through 7 Oct 2026). Placed in LLM preference optimization methods, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 33 of its 51 references that are in the map. First posted on arXiv 16 Jun 2024.

Builds on

33 of 51 references.

Closest ancestors · 9

Same direction, newest first

  1. 2024Is DPO Superior to PPO for LLM Alignment? A Comprehensive Study16 Apr
  2. 2024RS-DPO: A Hybrid Rejection Sampling and Direct Preference Optimization Method for Alignment of Large Language Models15 Feb
  3. 2024Aligner: Efficient Alignment by Learning to Correct4 Feb
  4. 2023A General Theoretical Paradigm to Understand Learning from Human Preferences18 Oct
  5. 2023Statistical Rejection Sampling Improves Preference Optimization13 Sep
  6. 2023Fine-Tuning Language Models with Advantage-Induced Policy Alignment4 Jun
  7. 2023Principle-Driven Self-Alignment of Language Models from Scratch with Minimal Human Supervision4 May
  8. 2022Training language models to follow instructions with human feedback4 Mar
  9. 2021A General Language Assistant as a Laboratory for Alignment1 Dec

Other references · 24

  1. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  2. 2017Proximal Policy Optimization Algorithmscited by 14,441
  3. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  4. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  5. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  6. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,916
  7. 2020Scaling Laws for Neural Language Modelscited by 6,411
  8. 2017Deep reinforcement learning from human preferencescited by 3,846
  9. 2023Qwen Technical Reportcited by 3,273
  10. 2022Scaling Instruction-Finetuned Language Modelscited by 2,997
  11. 2024Length-Controlled AlpacaEval: A Simple Way to Debias Automatic Evaluatorscited by 865
  12. 2023BeaverTails: Towards Improved Safety Alignment of LLM via a Human-Preference Datasetcited by 828
12 more
  1. 2023Enhancing Chat Language Models by Scaling High-quality Instructional Conversationscited by 825
  2. 2023Open Problems and Fundamental Limitations of Reinforcement Learning from Human Feedbackcited by 743
  3. 2023The Flan Collection: Designing Data and Methods for Effective Instruction Tuningcited by 734
  4. 2024Yi: Open Foundation Models by 01.AIcited by 706
  5. 2023RAFT: Reward rAnked FineTuning for Generative Foundation Model Alignmentcited by 672
  6. 2023Instruction Tuning for Large Language Models: A Surveycited by 617
  7. 2023RLAIF vs. RLHF: Scaling Reinforcement Learning from Human Feedback with AI Feedbackcited by 615
  8. 2023Weak-to-Strong Generalization: Eliciting Strong Capabilities With Weak Supervisioncited by 445
  9. 2023SLiC-HF: Sequence Likelihood Calibration with Human Feedbackcited by 395
  10. 2023Understanding the Effects of RLHF on LLM Generalisation and Diversitycited by 386
  11. 2023OpenChat: Advancing Open-source Language Models with Mixed-Quality Datacited by 250
  12. 2024Advancing LLM Reasoning Generalists with Preference Treescited by 191

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 2.

2 papers.

  1. 2026One Refiner to Unlock Them All: Inference-Time Reasoning Elicitation via Reinforcement Query Refinementcited by 6
  2. 2026Self-Correcting RAG: Enhancing Faithfulness via MMKP Context Selection and NLI-Guided MCTScited by 5

Similar, not cited

Close in wording, no citation either way

  1. 2024Direct Alignment of Language Models via Quality-Aware Self-Refinement
  2. 2025Self-Rewarding PPO: Aligning Large Language Models with Demonstrations Only
  3. 2025Evolution Strategies at Scale: LLM Fine-Tuning Beyond Reinforcement Learning
  4. 2024Aligning Large Language Models by On-Policy Self-Judgment
  5. 2025One-Token Rollout: Guiding Supervised Fine-Tuning of LLMs with Policy Gradient
  6. 2025SRFT: A Single-Stage Method with Supervised and Reinforcement Fine-Tuning for Reasoning
  7. 2026Bridging SFT and RL: Dynamic Policy Optimization for Robust Reasoning
  8. 2025Trust-Region Adaptive Policy Optimization