mlsanityData to 7 Oct 2026

Preference-Guided Reflective Sampling for Aligning Language Models

arXiv 2408.12163 ↗First posted 22 Aug 2024cs.CLcited by 4

On the map: LLM alignment / RLHF / preference optimisation → LLM preference optimization methods

Abstract (arXiv)

Iterative data generation and model re-training can effectively align large language models(LLMs) to human preferences. The process of data sampling is crucial, as it significantly influences the success of policy improvement. Repeated random sampling is a widely used method that independently queries the model multiple times to generate outputs. In this work, we propose a more effective sampling method, named Preference-Guided Reflective Sampling (PRS). Unlike random sampling, PRS employs a tree-based generation framework to enable more efficient sampling. It leverages adaptive self-refinement techniques to better explore the sampling space. By specifying user preferences in natural language, PRS can further optimize response generation according to these preferences. As a result, PRS can align models to diverse user preferences. Our experiments demonstrate that PRS generates higher-quality responses with significantly higher rewards. On AlpacaEval and Arena-Hard, PRS substantially outperforms repeated random sampling in best-of-N sampling. Moreover, PRS shows strong performance when applied in iterative offline RL training.

Key facts. Cited by 4 papers in the map (data through 7 Oct 2026). Placed in LLM preference optimization methods, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 24 of its 40 references that are in the map. First posted on arXiv 22 Aug 2024.

Builds on

24 of 40 references.

Closest ancestors · 2

Same direction, newest first

  1. 2023SALMON: Self-Alignment with Instructable Reward Models9 Oct
  2. 2023Principle-Driven Self-Alignment of Language Models from Scratch with Minimal Human Supervision4 May

Other references · 22

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2017Proximal Policy Optimization Algorithmscited by 14,441
  3. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  4. 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,355
  5. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,916
  6. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,634
  7. 2023Self-Refine: Iterative Refinement with Self-Feedbackcited by 3,317
  8. 2017Get To The Point: Summarization with Pointer-Generator Networkscited by 1,281
  9. 2023WizardLM: Empowering large pre-trained language models to follow complex instructionscited by 1,116
  10. 2022Red Teaming Language Models with Language Modelscited by 983
  11. 2024Large Language Monkeys: Scaling Inference Compute with Repeated Samplingcited by 946
  12. 2023Enhancing Chat Language Models by Scaling High-quality Instructional Conversationscited by 825
10 more
  1. 2023RAFT: Reward rAnked FineTuning for Generative Foundation Model Alignmentcited by 672
  2. 2023Reinforced Self-Training (ReST) for Language Modelingcited by 486
  3. 2024From Crowdsourced Data to High-Quality Benchmarks: Arena-Hard and BenchBuilder Pipelinecited by 484
  4. 2024Interpretable Preferences via Multi-Objective Reward Modeling and Mixture-of-Expertscited by 369
  5. 2023Alphazero-like Tree-Search can Guide Large Language Model Decoding and Trainingcited by 291
  6. 2017Toward Controlled Generation of Textcited by 271
  7. 2022Generating Sequences by Learning to Self-Correctcited by 271
  8. 2023Training Language Models with Language Feedback at Scalecited by 125
  9. 2023COLLIE: Systematic Construction of Constrained Text Generation Taskscited by 69
  10. 2024Benchmarking Large Language Models on Controllable Generation under Diversified Instructionscited by 26

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 1; 2025: 2; 2026: 1.

4 papers.

  1. 2025A Survey of Test-Time Compute: From Intuitive Inference to Deliberate Reasoningcited by 40
  2. 2025Finding the Sweet Spot: Preference Data Construction for Scaling Preference Optimizationcited by 9
  3. 2024Multi-Agent Sampling: Scaling Inference Compute for Data Synthesis with Tree Search-Based Agentic Collaborationcited by 8
  4. 2026Structuring the Space of Sociotechnical Alignmentcited by 2

Similar, not cited

Close in wording, no citation either way

  1. 2025Alignment through Meta-Weighted Online Sampling: Bridging the Gap between Data Generation and Preference Optimization
  2. 2025Clear Preferences Leave Traces: Reference Model-Guided Sampling for Preference Learning
  3. 2024West-of-N: Synthetic Preferences for Self-Improving Reward Models
  4. 2026How Sampling Shapes LLM Alignment: From One-Shot Optima to Iterative Dynamics
  5. 2024Towards a Unified View of Preference Learning for Large Language Models: A Survey
  6. 2024Active Preference Optimization for Sample Efficient RLHF
  7. 2024Investigating on RLHF methodology
  8. 2025HPS: Hard Preference Sampling for Human Preference Alignment