mlsanityData to 14 Oct 2026

Aligning Large Language Models through Synthetic Feedback

arXiv 2305.13735 ↗First posted 23 May 2023cs.CL cs.AI cs.LGcited by 75

On the map: LLM alignment / RLHF / preference optimisation → LLM preference optimization methods

Abstract (arXiv)

Aligning large language models (LLMs) to human values has become increasingly important as it enables sophisticated steering of LLMs. However, it requires significant human demonstrations and feedback or distillation from proprietary LLMs such as ChatGPT. In this work, we propose a novel alignment learning framework with synthetic feedback not dependent on extensive human annotations and proprietary LLMs. First, we perform reward modeling (RM) with synthetic feedback by contrasting responses from vanilla LLMs with various sizes and prompts. Then, we use the RM to simulate high-quality demonstrations to train a supervised policy and further optimize the model with reinforcement learning. Our resulting model, Aligned Language Model with Synthetic Training dataset (ALMoST), outperforms recent open-sourced models, which are trained on the outputs of InstructGPT or human-annotated demonstrations, in alignment benchmarks. In human evaluation, our model is preferred to Alpaca and Dolly-v2, 55.0% and 58.5% of the time, respectively. Further analyses demonstrate the efficacy and importance of synthetic feedback in our framework. The code is available at https://github.com/naver-ai/almost

Key facts. Cited by 75 papers in the map (data through 14 Oct 2026). Placed in LLM preference optimization methods, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 27 of its 41 references that are in the map. First posted on arXiv 23 May 2023.

Builds on

27 of 41 references.

Closest ancestors · 6

Same direction, newest first

  1. 2023LIMA: Less Is More for Alignment18 May
  2. 2023Principle-Driven Self-Alignment of Language Models from Scratch with Minimal Human Supervision4 May
  3. 2023OpenAssistant Conversations -- Democratizing Large Language Model Alignment14 Apr
  4. 2022Self-Instruct: Aligning Language Models with Self-Generated Instructions20 Dec
  5. 2022Training language models to follow instructions with human feedback4 Mar
  6. 2021A General Language Assistant as a Laboratory for Alignment1 Dec

Other references · 21

  1. 2020Language Models are Few-Shot Learnerscited by 32,419
  2. 2017Proximal Policy Optimization Algorithmscited by 15,215
  3. 2021Evaluating Large Language Models Trained on Codecited by 8,320
  4. 2020Measuring Massive Multitask Language Understandingcited by 7,963
  5. 2017Deep reinforcement learning from human preferencescited by 3,912
  6. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,720
  7. 2022Constitutional AI: Harmlessness from AI Feedbackcited by 3,050
  8. 2021TruthfulQA: Measuring How Models Mimic Human Falsehoodscited by 2,975
  9. 2019The Curious Case of Neural Text Degenerationcited by 2,916
  10. 2020Learning to summarize from human feedbackcited by 2,588
  11. 2019Fine-Tuning Language Models from Human Preferencescited by 2,150
  12. 2022Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language modelscited by 1,998
9 more
  1. 2023Pythia: A Suite for Analyzing Large Language Models Across Training and Scalingcited by 1,786
  2. 2021WebGPT: Browser-assisted question-answering with human feedbackcited by 1,745
  3. 2016The LAMBADA dataset: Word prediction requiring a broad discourse contextcited by 992
  4. 2023Instruction Tuning with GPT-4cited by 686
  5. 2022Improving alignment of dialogue agents via targeted human judgementscited by 531
  6. 2023RRHF: Rank Responses to Align Language Models with Human Feedback without tearscited by 463
  7. 2023Pretraining Language Models with Human Preferencescited by 247
  8. 2023Chain of Hindsight Aligns Language Models with Feedbackcited by 148
  9. 2023Training Language Models with Language Feedback at Scalecited by 128

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 19; 2024: 32; 2025: 19; 2026: 5.

75 papers.

Sort:

By area:

  1. 2023Open Problems and Fundamental Limitations of Reinforcement Learning from Human Feedbackcited by 759
  2. 2023A Comprehensive Overview of Large Language Modelscited by 648
  3. 2023Prometheus: Inducing Fine-grained Evaluation Capability in Language Modelscited by 575
  4. 2024A Survey on Knowledge Distillation of Large Language Modelscited by 284
  5. 2023Personalized Soups: Personalized Large Language Model Alignment via Post-hoc Parameter Mergingcited by 280
  6. 2023Large Language Model Alignment: A Surveycited by 243
  7. 2023FLASK: Fine-grained Language Model Evaluation based on Alignment Skill Setscited by 201
  8. 2025On Memory Construction and Retrieval for Personalized Conversational Agentscited by 105
  9. 2024A Review of Large Language Models and Autonomous Agents in Chemistrycited by 99
  10. 2025LLM Social Simulations Are a Promising Research Methodcited by 99
  11. 2023MemoChat: Tuning LLMs to Use Memos for Consistent Long-Range Open-Domain Conversationcited by 97
  12. 2024RLHF Deciphered: A Critical Analysis of Reinforcement Learning from Human Feedback for LLMscited by 90
38 more
  1. 2023EvalLM: Interactive Evaluation of Large Language Model Prompts on User-Defined Criteriacited by 74
  2. 2023SALMON: Self-Alignment with Instructable Reward Modelscited by 55
  3. 2023Self-Guard: Empower the LLM to Safeguard Itselfcited by 54
  4. 2025A Comprehensive Survey of Reward Models: Taxonomy, Applications, Challenges, and Futurecited by 53
  5. 2023Evaluating Instruction-Tuned Large Language Models on Code Comprehension and Generationcited by 50
  6. 2024Internal Consistency and Self-Feedback in Large Language Models: A Surveycited by 50
  7. 2023Learning from models beyond fine-tuningcited by 48
  8. 2023From Instructions to Intrinsic Human Values -- A Survey of Alignment Goals for Big Modelscited by 45
  9. 2023Efficient RLHF: Reducing the Memory Usage of PPOcited by 40
  10. 2024Surveying the Effects of Quality, Diversity, and Complexity in Synthetic Data From Large Language Modelscited by 35
  11. 2024LMUnit: Fine-grained Evaluation with Natural Language Unit Testscited by 33
  12. 2024How Far Are We From AGI: Are LLMs All We Need?cited by 31
  13. 2024West-of-N: Synthetic Preferences for Self-Improving Reward Modelscited by 28
  14. 2024ADELIE: Aligning Large Language Models on Information Extractioncited by 24
  15. 2024A Survey on Human Preference Learning for Large Language Modelscited by 24
  16. 2024Ask Optimal Questions: Aligning Large Language Models with Retriever's Preference in Conversationcited by 23
  17. 2023Data-Efficient Alignment of Large Language Models with Human Feedback Through Natural Languagecited by 22
  18. 2025CUPID: Evaluating Personalized and Contextualized Alignment of LLMs from Interactionscited by 22
  19. 2024On the Essence and Prospect: An Investigation of Alignment Approaches for Big Modelscited by 21
  20. 2025SelfCite: Self-Supervised Alignment for Context Attribution in Large Language Modelscited by 20
  21. 2024Evaluating Visual and Cultural Interpretation: The K-Viscuit Benchmark with Human-VLM Collaborationcited by 19
  22. 2024Position: Towards Bidirectional Human-AI Alignmentcited by 18
  23. 2024ROPO: Robust Preference Optimization for Large Language Modelscited by 16
  24. 2024Adaptive Query Rewriting: Aligning Rewriters through Marginal Probability of Conversational Answerscited by 15
  25. 2025A Survey on Progress in LLM Alignment from the Perspective of Reward Designcited by 15
  26. 2023CycleAlign: Iterative Distillation from Black-box LLM to White-box Models for Better Human Alignmentcited by 13
  27. 2024Dynamic Rewarding with Prompt Optimization Enables Tuning-free Self-Alignment of Language Modelscited by 13
  28. 2024Negating Negatives: Alignment with Human Negative Samples via Distributional Dispreference Optimizationcited by 11
  29. 2024Constraint Back-translation Improves Complex Instruction Following of Large Language Modelscited by 11
  30. 2023Unpacking the Ethical Value Alignment in Big Modelscited by 9
  31. 2024Boosting Reward Model with Preference-Conditional Multi-Aspect Synthetic Data Generationcited by 9
  32. 2024CBF-LLM: Safe Control for LLM Alignmentcited by 9
  33. 2025The Quest for Efficient Reasoning: A Data-Centric Benchmark to CoT Distillationcited by 9
  34. 2024Rethinking the Role of Proxy Rewards in Language Model Alignmentcited by 7
  35. 2024Bridging and Modeling Correlations in Pairwise Data for Direct Preference Optimizationcited by 7
  36. 2024Navigating Noisy Feedback: Enhancing Reinforcement Learning with Error-Prone Language Modelscited by 7
  37. 2025NEXUS: Network Exploration for eXploiting Unsafe Sequences in Multi-Turn LLM Jailbreakscited by 7
  38. 2024MoDiPO: text-to-motion alignment via AI-feedback-driven Direct Preference Optimizationcited by 6

50 most cited of 75 All 75 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025Aligning Large Language Models via Fully Self-Synthetic Data
  2. 2024Aligning Large Language Models via Fine-grained Supervision
  3. 2024The Real, the Better: Aligning Large Language Models with Online Human Behaviors
  4. 2024SAIL: Self-Improving Efficient Online Alignment of Large Language Models
  5. 2024DeAL: Decoding-time Alignment for Large Language Models
  6. 2024MetaAlign: Align Large Language Models with Diverse Preferences during Inference Time
  7. 2025Doubly Robust Alignment for Large Language Models
  8. 2025Align-Pro: A Principled Approach to Prompt Optimization for LLM Alignment