mlsanityData to 14 Oct 2026

Self-Boosting Large Language Models with Synthetic Preference Data

arXiv 2410.06961 ↗First posted 9 Oct 2024cs.CL cs.AIcited by 40

On the map: LLM alignment / RLHF / preference optimisation → LLM preference optimization methods

Abstract (arXiv)

Through alignment with human preferences, Large Language Models (LLMs) have advanced significantly in generating honest, harmless, and helpful responses. However, collecting high-quality preference data is a resource-intensive and creativity-demanding process, especially for the continual improvement of LLMs. We introduce SynPO, a self-boosting paradigm that leverages synthetic preference data for model alignment. SynPO employs an iterative mechanism wherein a self-prompt generator creates diverse prompts, and a response improver refines model responses progressively. This approach trains LLMs to autonomously learn the generative rewards for their own outputs and eliminates the need for large-scale annotation of prompts and human preferences. After four SynPO iterations, Llama3-8B and Mistral-7B show significant enhancements in instruction-following abilities, achieving over 22.1% win rate improvements on AlpacaEval 2.0 and ArenaHard. Simultaneously, SynPO improves the general performance of LLMs on various tasks, validated by a 3.2 to 5.0 average score increase on the well-recognized Open LLM leaderboard.

Key facts. Cited by 40 papers in the map (data through 14 Oct 2026). Placed in LLM preference optimization methods, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 53 of its 61 references that are in the map. First posted on arXiv 9 Oct 2024.

Builds on

53 of 61 references.

Closest ancestors · 9

Same direction, newest first

  1. 2024Better Alignment with Instruction Back-and-Forth Translation8 Aug
  2. 2024Direct Preference Knowledge Distillation for Large Language Models28 Jun
  3. 2024Spread Preference Annotation: Direct Preference Judgment for Efficient LLM Alignment6 Jun
  4. 2024Self-Augmented Preference Optimization: Off-Policy Paradigms for Language Model Alignment31 May
  5. 2024SimPO: Simple Preference Optimization with a Reference-Free Reward23 May
  6. 2024Scaling Data Diversity for Fine-Tuning Language Models in Human Alignment17 Mar
  7. 2022Self-Instruct: Aligning Language Models with Self-Generated Instructions20 Dec
  8. 2022Training language models to follow instructions with human feedback4 Mar
  9. 2021A General Language Assistant as a Laboratory for Alignment1 Dec

Other references · 44

  1. 2023GPT-4 Technical Reportcited by 18,396
  2. 2024The Llama 3 Herd of Modelscited by 14,715
  3. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  4. 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,170
  5. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
  6. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 8,065
  7. 2020Measuring Massive Multitask Language Understandingcited by 7,963
  8. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,886
  9. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,100
  10. 2023Self-Refine: Iterative Refinement with Self-Feedbackcited by 3,442
  11. 2022Constitutional AI: Harmlessness from AI Feedbackcited by 3,050
  12. 2021TruthfulQA: Measuring How Models Mimic Human Falsehoodscited by 2,975
32 more
  1. 2018Can a Suit of Armor Conduct Electricity? A New Dataset for Open Book Question Answeringcited by 2,164
  2. 2021WebGPT: Browser-assisted question-answering with human feedbackcited by 1,745
  3. 2023WizardLM: Empowering large pre-trained language models to follow complex instructionscited by 1,125
  4. 2018XNLI: Evaluating Cross-lingual Sentence Representationscited by 915
  5. 2024Length-Controlled AlpacaEval: A Simple Way to Debias Automatic Evaluatorscited by 897
  6. 2023Enhancing Chat Language Models by Scaling High-quality Instructional Conversationscited by 839
  7. 2023The RefinedWeb Dataset for Falcon LLM: Outperforming Curated Corpora with Web Data, and Web Data Onlycited by 763
  8. 2024Self-Rewarding Language Modelscited by 681
  9. 2019MathQA: Towards Interpretable Math Word Problem Solving with Operation-Based Formalismscited by 662
  10. 2023WizardMath: Empowering Mathematical Reasoning for Large Language Models via Reinforced Evol-Instructcited by 625
  11. 2023Textbooks Are All You Need II: phi-1.5 technical reportcited by 565
  12. 2023LLM-Blender: Ensembling Large Language Models with Pairwise Ranking and Generative Fusioncited by 564
  13. 2024Self-Play Fine-Tuning Converts Weak Language Models to Strong Language Modelscited by 557
  14. 2023Reinforced Self-Training (ReST) for Language Modelingcited by 500
  15. 2024From Crowdsourced Data to High-Quality Benchmarks: Arena-Hard and BenchBuilder Pipelinecited by 500
  16. 2022Unnatural Instructions: Tuning Language Models with (Almost) No Human Laborcited by 397
  17. 2024ReST-MCTS*: LLM Self-Training via Process Reward Guided Tree Searchcited by 344
  18. 2021Understanding Dataset Difficulty with V-Usable Informationcited by 291
  19. 2024Self-Play Preference Optimization for Language Model Alignmentcited by 232
  20. 2024Meta-Rewarding Language Models: Self-Improving Alignment with LLM-as-a-Meta-Judgecited by 204
  21. 2024Direct Nash Optimization: Teaching Language Models to Self-Improve with General Preferencescited by 178
  22. 2024Toward Self-Improvement of LLMs via Imagination, Searching, and Criticizingcited by 125
  23. 2024Synthetic Data (Almost) from Scratch: Generalized Instruction Tuning for Language Modelscited by 99
  24. 2024Self-playing Adversarial Language Game Enhances LLM Reasoningcited by 83
  25. 2022Describing Differences between Text Distributions with Natural Languagecited by 54
  26. 2021PROST: Physical Reasoning of Objects through Space and Timecited by 48
  27. 2023Won't Get Fooled Again: Answering Questions with False Premisescited by 48
  28. 2023Safer-Instruct: Aligning Language Models with Automated Preference Datacited by 27
  29. 2024Arena Learning: Build Data Flywheel for LLMs Post-training via Simulated Chatbot Arenacited by 26
  30. 2023SELF: Self-Evolution with Language Feedbackcited by 24
  31. 2024HRLAIF: Improvements in Helpfulness and Harmlessness in Open-domain Reinforcement Learning From AI Feedbackcited by 15
  32. 2024Towards Comprehensive Preference Data Collection for Reward Modelingcited by 5

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 4; 2025: 24; 2026: 12.

40 papers.

Sort:

By area:

  1. 2024LLMs-as-Judges: A Comprehensive Survey on LLM-based Evaluation Methodscited by 465
  2. 2025The Best Instruction-Tuning Data are Those That Fitcited by 52
  3. 2025First SFT, Second RL, Third UPT: Continual Improving Multi-Modal LLM Reasoning via Unsupervised Post-Trainingcited by 34
  4. 2024ProVision: Programmatically Scaling Vision-centric Instruction Data for Multimodal Language Modelscited by 21
  5. 2026Self-Play Only Evolves When Self-Synthetic Pipeline Ensures Learnable Information Gaincited by 17
  6. 2025Evolving LLMs' Self-Refinement Capability via Synergistic Training-Inference Optimizationcited by 16
  7. 2025What Matters in Data for DPO?cited by 12
  8. 2026Outcome Accuracy is Not Enough: Aligning the Reasoning Process of Reward Modelscited by 11
  9. 2025Finding the Sweet Spot: Preference Data Construction for Scaling Preference Optimizationcited by 9
  10. 2025From Generic Empathy to Personalized Emotional Support: A Self-Evolution Framework for User Preference Alignmentcited by 9
  11. 2025A Survey on Efficient Large Language Model Training: From Data-centric Perspectivescited by 9
  12. 2025RefineCoder: Iterative Improving of Large Language Models via Adaptive Critique Refinement for Code Generationcited by 8
28 more
  1. 2024Forewarned is Forearmed: Leveraging LLMs for Data Synthesis through Failure-Inducing Explorationcited by 7
  2. 2024CHAI for LLMs: Improving Code-Mixed Translation in Large Language Models through Reinforcement Learning with AI Feedbackcited by 5
  3. 2026Self-Boosting Vision-Language Models with Noisy Student On-Policy Self-Distillationcited by 4
  4. 2025Self-Improving Model Steeringcited by 3
  5. 2025Alignment through Meta-Weighted Online Sampling: Bridging the Gap between Data Generation and Preference Optimizationcited by 3
  6. 2025MapTrace: Scalable Data Generation for Route Tracing on Mapscited by 3
  7. 2025Data Swarms: Optimizable Generation of Synthetic Evaluation Datacited by 2
  8. 2025LLM-Driven Preference Data Synthesis for Proactive Prediction of the Next User Utterance in Human-Machine Dialoguecited by 2
  9. 2026Reflect: Transparent Principle-Guided Reasoning for Constitutional Alignment at Scalecited by 2
  10. 2025SGPO: Self-Generated Preference Optimization based on Self-Improvercited by 1
  11. 2025GRAPH-GRPO-LEX: Contract Graph Modeling and Reinforcement Learning with Group Relative Policy Optimizationcited by 1
  12. 2025AutoMalDesc: Large-Scale Script Analysis for Cyber Threat Researchcited by 1
  13. 2025Learning from Self Critique and Refinement for Faithful LLM Summarizationcited by 1
  14. 2025Fine-Tuning LLMs with Fine-Grained Human Feedback on Text Spanscited by 1
  15. 2026Generating Leakage-Free Benchmarks for Robust RAG Evaluationcited by 1
  16. 2025Amulet: Putting Complex Multi-Turn Conversations on the Stand with LLM Juriescited by 0
  17. 2025RACE-Align: Retrieval-Augmented and Chain-of-Thought Enhanced Preference Alignment for Large Language Modelscited by 0
  18. 2025From Threat to Tool: Leveraging Refusal-Aware Injection Attacks for Safety Alignmentcited by 0
  19. 2025Icon2: Aligning Large Language Models Using Self-Synthetic Preference Data via Inherent Regulationcited by 0
  20. 2025SeaPO: Strategic Error Amplification for Robust Preference Optimization of Large Language Modelscited by 0
  21. 2025Revisiting Self-Play Preference Optimization: On the Role of Prompt Difficultycited by 0
  22. 2026The Subjectivity of Respect in Police Traffic Stops: Modeling Community Perspectives in Body-Worn Camera Footagecited by 0
  23. 2026Easy Samples Are All You Need: Self-Evolving LLMs via Data-Efficient Reinforcement Learningcited by 0
  24. 2026Towards Human-Level Book-Writing Capabilitycited by 0
  25. 2026Vision-driven Preference Synthesis for Mitigating Hallucinations in VLMscited by 0
  26. 2026Positive-Unlabeled Preference Optimization For Chest X-ray Report Generationcited by 0
  27. 2026Instruction Quality Matters: Refining Instructions for Effective Preference Learningcited by 0
  28. 2026Too Good to Be Real? Diagnosing and Reducing the Gap Between AI Preference and Real User Engagementcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Aligning Large Language Models via Fully Self-Synthetic Data
  2. 2024Optimizing LLMs with Direct Preferences: A Data Efficiency Perspective
  3. 2024Relative Preference Optimization: Enhancing LLM Alignment through Contrasting Responses across Identical and Diverse Prompts
  4. 2025Beyond Pairwise: Empowering LLM Alignment With Ranked Choice Modeling
  5. 2025LongPO: Long Context Self-Evolution of Large Language Models through Short-to-Long Preference Optimization
  6. 2025InfoPO: On Mutual Information Maximization for Large Language Model Alignment
  7. 2024Refined Direct Preference Optimization with Synthetic Data for Behavioral Alignment of LLMs
  8. 2024Unified Preference Optimization: Language Model Alignment Beyond the Preference Frontier